Making Gemma generate text faster
How changes to token generation and GPU execution affected speed in a Gemma benchmark.
Making a language model generate faster is useful only if the quality of its output remains acceptable. This experiment compared two Gemma inference configurations using both speed and perplexity, a measure related to how well a model predicts text.
How the faster configuration worked
The experiment combined several ways to reduce the work and waiting involved in generation. Speculative decoding proposes possible next tokens so the model can check them efficiently. CUDA graph replay reduces repeated setup work on the GPU by replaying a prepared sequence of operations.
The adaptive configuration also selected among candidate counts of 32, 40, 48 and 64 using a policy that balanced trying options with using those that had performed well. It built on an existing inference stack and included a cache fix, so the recorded result reflects the combined configuration rather than the selection policy alone.
The number of candidates can affect how much work is done before the model settles on the next part of its output. Trying different counts allows the configuration to adapt that choice. The benchmark compared the result of the whole approach with the earlier fixed configuration.
Reading the speed and quality measures
Tokens per second measures how quickly the model generates pieces of text. Tokens are not always whole words, but the measure is useful for comparing configurations under the same benchmark. Perplexity measures how surprised a model is by the test text: lower is generally better for this measure, though it is not a complete assessment of answer quality.
Both configurations completed all 128 benchmark items. The earlier configuration generated about 304 tokens per second. The adaptive configuration generated about 311. Their perplexity scores were very close, with a small increase in the faster version. The table gives the values rounded for easier comparison.
The perplexity values are close, but they describe a particular test rather than every kind of response a model might produce. Keeping the measure next to throughput makes the trade visible instead of treating speed as the only outcome.
| Configuration | Tokens per second | Perplexity |
|---|---|---|
| Earlier fixed configuration | 304.38 | 2.0266 |
| Adaptive configuration | 311.40 | 2.0269 |
Separating the contributions
Because several changes were present together, this comparison cannot say how much each one contributed. Repeating the benchmark and changing one component at a time would help distinguish their effects.
These are historical results for the recorded setup. The comparison illustrates a speed and perplexity trade-off within that benchmark, rather than establishing a current leaderboard position or a speedup on every machine.