How we ran 64 AI agents on a 6 GB laptop GPU
How thinking credits, selective reasoning and memory techniques helped manage limited computing capacity.
A village with many AI characters must decide how to share a limited amount of computing power. We tried sending every decision to the larger Gemma model. With 64 villagers on a laptop GPU with 6 GB of memory, that run used its time budget before the simulated day was halfway over. We then tested a way to reserve deeper reasoning for selected decisions.
Sharing a limited budget
We tested villages of 8, 16, 32 and 64 villagers, allowing each run 900 seconds. Both decision policies used the same backend queue, with room for four requests running and sixteen waiting. The additional villagers were copies of the original eight, with houses on the same land. Sixty-four was the largest village tested.
In the 64-villager test, the policy that always used Gemma reached about 12:08 after 901.1 seconds. The selective policy completed a simulated day from 06:00 to 22:00 in 421.9 seconds, making 628 decisions and 40 Gemma calls. The runs followed different trajectories and reached different endpoints, so those times do not give a matched whole-day speedup.
The challenge grows with the number of villagers because many characters can need decisions at once. A queue can hold only so much work, and requests to a larger model take time. When all decisions use that model, routine choices compete for the same capacity as emergencies. The selective policy is an attempt to manage that competition.
Saving attention for a busy moment
The selective policy uses a shared bucket of thinking credits. Each real decision opportunity adds 0.08 credit. Asking Gemma to consider one villager's decision costs one credit. The bucket starts empty, can hold up to 24 credits and can spend at most four at a time. Idle frames do not earn credits.
During quiet periods, unused credits can build up for a burst of difficult decisions. Because the bucket starts empty, the total number of decisions sent for deeper reasoning cannot exceed 8% of decision opportunities. Backend capacity also limits how many model requests can run at once.
In a separate eight-villager fire scenario, always using Gemma took 176.3 seconds and 71 calls. Selective routing took 24.6 seconds and six calls. Both runs put out the fire without recorded house damage or hunger, but the selective run had lower mean reward. It used less time and fewer calls with a measured trade-off in the overall outcome.
Choosing when to ask
A router is the component that decides whether a villager should keep Julia's quick choice or ask Gemma to deliberate. This router uses a small statistical model to estimate the immediate benefit of asking. It considers information available before the decision, including how close the quick model's scores are, the stakes, surprise and the type of action.
On a replay of 144 held-out decisions, the router called Gemma 14 times and achieved 105.05% of the always-Gemma reward score. The estimated interval ranged from 97.93% to 113.34%. Results varied by situation: the fire subset stayed below 95%, and live full-day comparisons had lower mean reward. The replay demonstrates a way to allocate calls, while leaving room to improve when the router asks for help.
The router cannot see Gemma’s answer or the eventual outcome when it makes its choice. It must estimate the value of asking from the current situation. This is why a routing policy can save calls but still miss cases where extra reasoning would have helped. The fire results make that limitation concrete.
Finding memories without scoring every pair
To find a useful memory, the system compares a numerical description of the current situation with numerical descriptions of past events. These descriptions are called vectors. Comparing every possible pair can create a lot of work, so one technique starts with a shortlist and calculates only the scores needed for that list.
Villagers also need to retrieve memories. One technique computes similarity scores only for pairs already on a shortlist. In a JavaScript benchmark, calculating 16,000 requested scores took 1.1154 milliseconds, while calculating a full table of one million scores took 74.5348 milliseconds. The requested scores matched. The timing did not include building the shortlist, so it describes this calculation rather than the whole retrieval process.
Keep a small index and check the originals when needed
A separate memory technique stores compressed numerical keys to narrow down which memories might rank highest. When those compressed values cannot settle the order, it reads the original, more precise vectors and scores them again. All 256 tested recall replies matched the reference scoring method. The compressed setup used 310,400 bytes in memory compared with 1,228,800 bytes for the original representation, and read an average of 17.72 of 400 original vectors per query.
The original float32 method was still faster on this data and remains the default. Matching the scoring method means the same memories are ranked, not that those memories are necessarily true or useful. In the worst case the compressed method must read every original vector.
A memory shortcut that stayed off
We also tried a more aggressive way to compress memory. On its small test set, its top three results overlapped with the true top three only 67.6% of the time. That version stayed disabled.
The village combines several separate ideas: a budget limits requests for deeper reasoning, a router chooses which decisions may benefit, and memory methods manage retrieval work. Each solves a different part of the capacity problem, and each needs to be measured on its own as well as in the full simulation.