The Model Card's Recommended Config Was 1.9x Slower Than No Config

I ran NVIDIA’s Nemotron 3.5 Lightning 30B-A3B (the NVFP4 checkpoint) on a DGX Spark, following the model card’s own quick-start recipe for that exact machine. Then I removed one thing from the recipe — the speculative decoding it recommends — and throughput went from about 41 tok/s to 77.5.

The vendor’s recommended configuration for the vendor’s own hardware was a 1.9x regression. Not a subtle tuning miss: the single flag the recipe leads with is the one costing half the tokens.

The useful part isn’t that a recipe was wrong. It’s how it was wrong, and how convincing the wrong configuration looked from the inside — the drafter was working, acceptance rates were healthy, and every metric I’d normally check said the feature was earning its keep.

The setup

Machine	DGX Spark, GB10 Grace Blackwell, aarch64, sm_121a
Memory	~121 GiB unified CPU/GPU, ~273 GB/s (LPDDR5x)
Engine	vLLM 0.27.1, PyTorch 2.13.0+cu130, CUDA 13.0
Model	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, 21 GB
Draft model	the companion DSpark checkpoint, 1.3 GB

The model is a hybrid MoE: 52 layers alternating Mamba-2 and MoE blocks with a full-attention layer roughly every sixth position — six attention layers in the whole stack. 30B total parameters, ~3B active per token, 128 routed experts with 6 active plus one shared expert. Quantisation is NVFP4 in a “four over six” arrangement: W4A16 on the routed and shared expert weights, FP8 per-tensor dynamic scales on the Mamba projections and the KV cache. Context goes to 1M tokens.

Two things about the install are worth recording because they were the opposite of the last hybrid model I stood up on this box, which needed source builds and several patches.

vLLM ships official aarch64 + CUDA 13 wheels for 0.27.1. pip install vllm==0.27.1 into a fresh venv pulled a manylinux_2_28_aarch64 wheel and that was the entire install. No source build of vLLM, no FlashInfer build, no Marlin or Mamba/MoE kernel compilation — for a brand-new architecture on brand-new hardware. That is not where this usually lands and it deserves saying out loud.

I checked every flag in the recipe before running it. The card’s DGX Spark quick-start specifies --moe-backend marlin, --mamba-backend flashinfer, --kv-cache-dtype fp8, --reasoning-parser nemotron_v3, --tool-call-parser qwen3_coder and more. I cross-checked all of them against this install’s vllm serve --help=all, against ModelRegistry, and against the reasoning-parser directory. All present, all correctly named. These recipes rot fast across engine releases and a stale flag name produces a confusing failure ten minutes into a load — the five minutes to verify is always worth it.

The one correction: --speculative-config takes nested JSON, not the card’s literal --speculative_config.model dot-flags. Those do work through vLLM’s dataclass-field expansion, but the plain JSON string is more robust:

--speculative-config '{"model": "<draft-path>", "num_speculative_tokens": 3}'

Method

A small script hits the running server’s /v1/chat/completions with stream: true and measures time-to-first-token plus post-TTFT decode rate from the SSE event stream. enable_thinking: false throughout, so reasoning tokens don’t contaminate the decode number. Single stream, batch size 1 — the “one person typing” number. Two prompts, a code one at ~250 completion tokens and a long-form one at 300, three runs each per configuration. Acceptance rates come from vLLM’s own Prometheus counters (vllm:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total), not from anything I computed.

Results

Config	Decode tok/s	Draft accept rate
Model card recipe: DSpark, num_speculative_tokens=3	~41	53.7%
DSpark, num_speculative_tokens=4	27–34	45.2%
DSpark, num_speculative_tokens=2	~40	62.4%
No speculative decoding	~77.5	—
No spec-decode, explicit --mamba-ssu-algorithm horizontal	~77.5	—
--async-scheduling	no effect at batch 1	—

Every speculative configuration loses to no speculative configuration, by a wide margin and regardless of draft depth.

Why it loses

vLLM says it in the boot log, in plain language, and I had read past it twice:

CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention
backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE);
setting cudagraph_mode=PIECEWISE

With speculative decoding on, decode steps run through a partially-captured PIECEWISE graph instead of the fully-captured FULL_AND_PIECEWISE path that plain autoregressive decode gets. The drafter is doing its job — 53.7% acceptance on 3-token drafts is roughly 1.6 free tokens per verify step, which should be a solid win — but the per-step cost of losing full graph capture is larger than the tokens it buys. Net 1.9x loss.

The draft depth can’t rescue it, and looking at draft depth is the trap. Dropping to num_speculative_tokens=2 gets a much better acceptance rate — 62.4% against 53.7%, which is what you’d expect since shorter proposals are easier to get right — and it is still slower than not speculating at all, because the graph-capture penalty is paid per step regardless of how long the draft is. Going to 4 is worse on both axes at once: acceptance falls to 45.2% as the draft wanders further from the current token, and the verify step costs more. Three, the card’s default, is genuinely the best of the three speculative settings. It is not the best setting.

That is the shape of the mistake, and it’s the transferable part: I spent the tuning pass optimising acceptance rate, which is a metric that moves independently of the thing I actually wanted. Acceptance rate told me the drafter was good. It could not tell me that enabling the drafter had changed the execution mode of every step it wasn’t involved in. The only measurement that catches this is end-to-end throughput against the feature switched fully off — a control I nearly didn’t run, because why would you benchmark the configuration the vendor tells you not to use.

I’d expect this to be specific to this engine version’s CUDA-graph and spec-decode interaction on FlashInferBackend, not a fact about speculative decoding. The acceptance rates say it should be a clear win once the graph penalty goes away. It’s worth re-testing on every vLLM release, and I intend to.

The backend built for this chip doesn’t work yet

vLLM has flashinfer_b12x, a backend path specifically for SM12x — this chip. Both ways of enabling it failed, and the failures are worth recording because they’re incompatibilities rather than flag-tuning problems.

With --linear-backend flashinfer_b12x:

ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x'
kernel exists for this layer type

That backend wants native FP4 activations. This checkpoint is W4A16 — 4-bit weights, 16-bit activations — so the shared linear layers have no matching kernel. A checkpoint property, not a configuration one.

With --moe-backend flashinfer_b12x alone, leaving linear layers on auto:

RuntimeError: shape '[344064, 116]' is invalid for input of size 41287680

A reshape bug hit by this checkpoint’s expert layout (128 routed experts plus shared-expert overlap). vLLM flags the backend experimental in its own boot warnings, so this is a known frontier rather than a surprise.

Sparsity is the whole story on this machine

Three days ago I benchmarked a dense 27B on this same box and got 4.57 tok/s at BF16, because every decode step has to pull all 54 GB of weights through a 273 GB/s pipe. I argued in that post that an MoE would change the picture completely, ran the arithmetic, and was explicit that it was reasoning and not a measurement.

This is the measurement. Same box, same bandwidth, 77.5 tok/s against 4.57 — a model that is larger on paper running 17x faster, because it activates ~3B of its 30B parameters per token and those parameters are 4-bit. Capacity gates what you can load; sparsity decides whether the bandwidth number matters. The Spark looks hopeless on a dense model and entirely reasonable one architecture over, which is what the machine is for.

The roof arithmetic still leaves room. ~3B active parameters at 4 bits is roughly 1.5 GB of expert weights streamed per decode step; at 273 GB/s that’s ~5.5 ms per token, or about 182 tok/s from routed-expert weight movement alone. Measured 77.5 is well under that.

Two observations are consistent with the gap being kernel-bound rather than bandwidth-bound: GPU utilisation sat at 94% while power draw stayed around 32 W, and clocks held at 2405/3003 MHz the whole time — not thermally throttled, not doing much arithmetic either. That profile says kernel launch and dequantisation, not compute. A working flashinfer_b12x, or a checkpoint with native FP4 activations, is where I’d look to close it.

That last paragraph is extrapolation, not a result. The 182 tok/s figure ignores Mamba layer traffic, the shared expert, KV and SSM state reads, and routing overhead. It is the reason I think there’s headroom, not evidence of how much.

What’s unproven

Everything here is batch 1. Speculative decoding’s arithmetic can invert under concurrent load, where the GPU is already busy and the verify step’s extra work rides along in slack that exists anyway. I have not ruled DSpark out for multi-user serving; I’ve ruled it out for one person typing on this engine version. Those are different claims and only the second one is mine.

I did not test --kv-cache-dtype nvfp4, which vLLM accepts and which is more aggressive than the fp8 KV cache used throughout. Unclear accuracy and speed tradeoff, untried.

The three-runs-per-configuration protocol is thinner than the repeat-everything discipline I used on the last benchmark. The gaps here are large enough — 77.5 against 41 — that run noise doesn’t threaten the conclusion, but the 40 vs 41 ordering between num_speculative_tokens 2 and 3 is inside the noise and I’m not claiming it.

And, as with every benchmark I’ve published on this hardware: I measured tokens per second and nothing whatsoever about output quality. NVFP4 is aggressive quantisation and none of these numbers say the model is still good. That evaluation keeps being the next thing to run.