The DSpark Draft Model Is a Model Choice, Not a Switch

Speculative decoding gets discussed as a feature you turn on. Pick an algorithm, point the server at a draft checkpoint, set a block size, measure the speedup. I did exactly that a few days ago and got Qwen3.8-27B from 82.9 tok/s to 159.1 on a single RTX 5090.

Then I swapped the draft checkpoint for a different one — same algorithm, same block size, same target, every other server flag byte-identical — and got 176.8 tok/s. An 11.3% gain from changing nothing but which drafter I pointed at.

The drafter is a model you choose, with its own quantisation, its own training corpus, and its own memory footprint that competes with your KV cache. It deserves the same scrutiny you give the target. Almost nobody gives it that.

The setup

This continues from the Qwen3.8-27B benchmark, which ended at 159.1 tok/s after three fixes: a newer SGLang, a pure-NVFP4 target checkpoint, and switching from the model’s built-in MTP head to DSpark with a block size of 7.

Same box and same protocol as before: one RTX 5090, SGLang’s single-batch benchmark, batch size 1, 1,024 input tokens, 256 output, single stream. The target is a 4-bit NVFP4 build of Qwen3.8-27B with lm_head left unquantised — DSpark drafters ship no output head of their own and borrow the target’s, so a checkpoint that packs lm_head to FP4 fails to load at all.

The only thing that varies below is the draft checkpoint.

Two candidates

Drafter	Precision	On disk	Resident
Stock DSpark	BF16	2.6 GB	2.71 GB
NVFP4 DSpark	mixed 4-bit	1.4 GB	1.45 GB

Those are RadixArk/Qwen3.8-27B-DSpark and gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4, both drafting for a gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 target.

The second is not simply the first one quantised. It is a different checkpoint that was both retrained and quantised, which matters for reading the result and I’ll come back to it.

The numbers

Each configuration ran twice, cold, minutes apart:

Drafter	Run 1	Run 2	Accepted length	KV pool (tokens)
No speculation	82.93	82.91	—	—
Built-in MTP head	118.17	118.04	1.93	—
DSpark, BF16	158.70	158.97	3.01	34,377
DSpark, NVFP4	176.72	176.87	3.15	55,285

Run-to-run spread is under 0.2%, and the BF16 row reproduces the 159.14/158.89 I measured on a different day, so the harness is stable enough to trust an 11% difference.

Against no speculation at all, the 4-bit drafter is 2.13x.

The part I didn’t expect

Quantising a draft model to 4 bits should make it a worse predictor. A worse predictor proposes tokens the target rejects more often, acceptance falls, and you claw back some wall-clock because each draft pass is cheaper. That’s the trade I went in expecting: faster drafting, lower acceptance, net gain if you’re lucky.

Acceptance went up. 3.01 to 3.15 tokens per verification step.

So the 4-bit drafter is both cheaper to run and better at guessing. That combination is why the gain is 11% rather than the two or three I’d have guessed.

I can’t cleanly attribute it, and neither should you. The two checkpoints differ in two ways at once — precision and training data. The NVFP4 one was retrained on a corpus regenerated through the target’s actual chat template, closing a mismatch where the old drafter had been trained on tool-call formatting the server never emits. Quantisation and retraining shipped together in one release. My benchmark varies both simultaneously and cannot say which one bought the acceptance.

The honest reading is narrower than the headline: this drafter beats that drafter by 11.3% on this workload. Not “quantising drafters is free.”

The publisher’s own card claims +3.3% over the BF16 drafter, measured on 240 held-out prompts at 64K context. I measured +11.3% on 1,024-token synthetic prompts at batch 1. The direction agrees and the magnitude doesn’t, which is roughly what you’d expect from two workloads this different. If your workload looks like theirs, believe their number over mine.

The memory dividend nobody mentions

The drafter is resident in VRAM for the life of the server, and on a 32 GB card every gigabyte it holds is a gigabyte the KV cache doesn’t get.

Dropping from 2.71 GB to 1.45 GB freed 1.26 GB. At an unchanged --mem-fraction-static 0.90, the token pool went from 34,377 to 55,285 tokens — 61% more context, for free, as a side effect of a change I made for throughput.

On a card where the model barely fits, that is arguably the bigger result. Throughput gains of 11% are pleasant; a 61% larger context window changes which requests you can serve at all. It also doesn’t show up in any throughput benchmark, which is why I nearly missed it — I only looked because the server log prints max_total_num_tokens at startup and the number had moved.

One flag is not optional. The checkpoint ships pre-quantised, and without --speculative-draft-model-quantization modelopt_fp4 the engine infers the drafter’s quantisation from the target’s setting and the load fails outright. It fails loudly, which is the good kind of failure, but it’s not inferred for you.

What this doesn’t establish

The prompts are synthetic — random token ids, which is what the benchmark harness generates. Acceptance length on random tokens is a real measurement of a strange workload. A drafter tuned on natural text and tool calls may well rank differently on prose than it does here, and the per-domain tables on the model card show acceptance varying from 2.3 to 4.4 across domains. My single number averages over none of that.

Everything is batch 1, single stream. This is the “one person typing” number. Speculation economics change completely under concurrency, where the spare compute the drafter is exploiting has already been claimed by other requests.

And I tested two drafters out of a possible many, on one target, on one GPU.

What I’d defend: on this box, for single-stream decode, the draft checkpoint was worth 11% throughput and 61% context, and it was the highest-leverage change available after the obvious ones were done. It cost one download and one flag. If you’re running speculative decoding and you accepted whichever drafter the first tutorial handed you, that’s probably the cheapest experiment on your list.