Tokens per second —
for your GPU.
We pushed 8 local models — from 9B to 80B — across context windows from 64K to 262K on a single RTX 4090, and logged every result as tokens per second (TPS): the one number that tells you how a GPU actually performs under real inference load.
The fastest tokens per second in the test
Top results across every model, quantization, and context size — sorted by raw tokens-per-second (TPS) throughput.
Eight models, one card slot
From the frugal 9B to the 80B heavyweight — here's how every tested model behaves on a single 24GB card.
The context cliff
Qwen 3.6 35B A3B at full GPU offload, across rising context windows — the point where VRAM pressure makes performance collapse.
Speed vs. Context Window
qwen3.6-35b-a3b · full GPU offload · tok/sWhat the numbers actually show
Four takeaways that matter for any 24GB setup — not just these particular models.
Full offload backfires on MoE models
For models with few active parameters (A3B, A4B), 100% GPU offload often slows things down through swapping — more than targeted partial offload or CPU-forced MoE layers would.
144K is the sweet spot
For Qwen 3.6 35B A3B, a context window around 144K delivers the best balance of memory footprint and tokens/second — beyond that, things get unstable.
80B is a bridge too far
Qwen 3 Coder Next 80B A3B needs over 46GB of combined GPU and shared memory even at just 64K context — practically unusable on 24GB.
Quantization saves runs
Qwen 3 Coder 30B: 4.86 tok/s at full precision → 96.88 tok/s with Q4_0. Same context, 20x faster.