MTP Doubled Local Throughput and Quietly Cost Me 64K of Context

I upgraded llama.cpp on my workstation for one reason: I wanted to know what enabling MTP would actually do to inference speed.

The model I run locally is a 27B Qwen3.6 community finetune, quantised to Q6_K, on a single 32 GB consumer card. It ships in two variants — one with a multi-token prediction head and one without. MTP is a form of self-speculative decoding: the model carries an extra block that predicts several tokens ahead, and the main model then verifies that guess in a single forward pass. When the guess is right, you get multiple tokens for roughly the cost of one. No separate draft model required.

I had never been able to use it. The build I was running treated the MTP head’s extra block as if it were a regular transformer layer and refused to load the file at all:

missing tensor 'blk.64.ssm_conv1d.weight'

So my server scripts pointed at the non-MTP file and approximated the same idea with n-gram drafting — a cheap heuristic that guesses the next few tokens from repeated patterns in the context, needing no model support at all. It was the stand-in, and I’d left a comment in the script saying so.

The new build added a real draft-mtp speculative decoding type. This post is what happened when I measured it.

The headline number

Same prompt, greedy decoding, 180 tokens:

Configuration	Tokens/sec	Drafts accepted
Non-MTP model + n-gram drafting	58.1	0 of 0
MTP model + draft-mtp	136.3	73 of 87 (84%)

2.3x. On short generations in the finished config I saw 219 tok/s with a 78% acceptance rate, and 124 tok/s with the vision encoder loaded.

The zero in that first row is the part worth sitting with. N-gram drafting didn’t lose the race — it never entered it. On a prompt asking the model to count from one to forty in words, which is about as repetitive as generated text gets, it produced no accepted drafts at all. I had been running a speculative decoding strategy that, for my actual workload, was doing nothing. The 58 tok/s baseline is just the model decoding normally with some wasted overhead on top.

That’s the first transferable bit: a speculative decoding setting that is enabled is not the same as one that is working. Both configs reported speculative decoding as on. Only one of them was drafting. If your inference server exposes draft acceptance counts, look at them — a 0% acceptance rate is invisible in throughput terms if you have no faster configuration to compare against. I’d been running the useless version for weeks and had no signal that anything was wrong.

The MTP head, by contrast, drafts against the model’s own learned continuation, so it tracks whatever the model is about to say rather than whatever it has already said. 84% on counting, 78% on prose, 76% on an image description. It holds up across content types in a way the n-gram heuristic obviously doesn’t.

The part I wasn’t looking for

Having measured the win, I went to update my two server scripts — one with a vision encoder at 200K context, one text-only at 262K — and neither would start. Out of memory allocating a 1.3 GB compute buffer.

The MTP file is larger than the non-MTP one, but only by 0.42 GB, which didn’t come close to explaining it. So I tested the control: the old model with the old flags, the exact command line that had worked before the upgrade.

It also OOMed.

The upgrade itself had increased VRAM usage enough to break a working configuration, independent of anything to do with MTP. My text-only script had been broken by the git pull, and if I hadn’t happened to be benchmarking that week I’d have discovered it the next time I tried to start the server — with no obvious reason to suspect the upgrade, since the failure is an allocator error rather than anything mentioning a version.

I tried the obvious mitigation first. The failure names a compute buffer, and compute buffer size scales with micro-batch size, so I dropped --ubatch-size to 256 and --batch-size to 1024 and retried the full 262K. Still OOM. The compute buffer was where the allocation happened to fail, not what was actually consuming the memory — the KV cache had grown, and no batch tuning was going to give that back.

Finding the real ceiling

Rather than guess a safe number, I bisected it. A small script that starts the server at a given context length, watches the log for either a listening line or an allocator failure, records peak VRAM, and kills it:

Context	Text-only	With vision encoder
262,144	OOM	—
245,760	OOM	—
229,376	OOM	—
212,992	OOM	—
196,608	fits — 31741 MiB	OOM
180,224	—	fits — 31923 MiB
163,840	—	fits — 31469 MiB

Final numbers: 262,144 to 196,608 text-only, and 200,000 to 163,840 with vision. Roughly a 25% cut in both cases, in exchange for a bit over 2x throughput. For my use that’s an easy trade, but it is a trade, and I would not have known I was making it if I’d only measured the speed.

I took 163,840 for the vision config rather than the 180,224 that technically loads. That row fits with about 680 MiB of headroom on a 32 GB card, which is not enough margin to trust when a long prompt starts allocating. Verified peak on a real 40,010-token prompt was 31783 MiB against a 32607 MiB card — under 900 MiB spare even at the size I did pick. Load success is a weak signal here; peak usage under a realistic prompt is the number that matters.

The flag I almost broke things with

One more thing worth writing down, because it was the closest I came to shipping a silent regression.

My scripts preserve the model’s reasoning traces across turns using a template keyword argument:

--chat-template-kwargs '{"preserve_thinking": true}'

The new build adds what looks like a purpose-built replacement, --reasoning-preserve. Cleaner, first-class, obviously the modern way to express the same intent. I wrote it into both scripts, and the server started fine with it.

Then I read the flag’s own help text, which says it applies to templates advertising a supports_preserve_reasoning capability. So I pulled the model’s chat template from the server’s /props endpoint and grepped it:

has preserve_thinking in template:        True
has supports_preserve_reasoning:          False

The template branches on preserve_thinking. It does not declare the capability the new flag keys off. The flag would have been accepted, logged nothing, and done nothing — reasoning preservation would have quietly stopped working, and the failure mode is degraded multi-turn output rather than an error.

I reverted to the keyword-argument form and left a comment in both scripts explaining why, because it looks exactly like dead code that a future cleanup pass would delete.

A new flag that supersedes an old workaround is only an improvement if the model you’re running opts into it. Server accepted it, server started, server was ignoring it.

What I’d generalise

Three things, none of them specific to this model or this inference server:

• Verify the mechanism fired, not just that it was enabled. Draft acceptance counts turned “speculative decoding is on” into “speculative decoding is doing nothing,” and those look identical from throughput alone.
• When an upgrade breaks something, test the old configuration on the new build before blaming your changes. I spent a while assuming the larger MTP file was the problem. One control run pointed at the upgrade instead, which is a different fix and a different thing to write down.
• Measure the cost side of a speed win. Both scripts got faster and smaller in the same edit. Only one of those was the thing I set out to change.

The unproven part: I benchmarked throughput and acceptance rates, not output quality. MTP verification is supposed to be lossless — accepted drafts are exactly the tokens the model would have produced anyway — but I did not test that empirically, and “supposed to be lossless” is doing real work in that sentence. The outputs I looked at were coherent and on-topic. That is not the same as verifying token-level equivalence against a non-drafted run, which is the experiment I should do next and haven’t.