Qwen3.8-27B on a DGX Spark and an RTX 5090: Capacity Gets You In, Bandwidth Decides Everything Else

I benchmarked Qwen3.8-27B, with the same harness and the same workload, on two machines that fail in opposite directions: an NVIDIA DGX Spark and an RTX 5090. The Spark has four times the memory and roughly a sixth of the bandwidth. The 5090 cannot hold the model at any precision above 4-bit.

The 4-bit run on the 5090 is 12x faster than the full-precision run on the Spark. That part is unsurprising, and it is the part that holds up.

What I originally wrote next — that the 5090 sits much further from its own theoretical ceiling, so past some speed the constraint changes character — did not survive contact with a better software stack. A day after publishing, SGLang reported 206.1 tok/s for this model on this GPU against my 54.6, and chasing that gap turned up three fixable problems on my side and one wrong conclusion in this post. Both the original analysis and the correction are below, in that order; the numbers I first published are unchanged and still reproducible under the configuration stated.

Two machines

	DGX Spark (GB10)	RTX 5090
Memory	~121 GiB unified CPU/GPU	31.4 GiB dedicated
Bandwidth	~273 GB/s (LPDDR5x)	~1,792 GB/s (GDDR7, spec)
Arch	aarch64, sm_121	x86-64, sm_120
Largest variant that fits	BF16 (52 GB)	NVFP4 (21 GB)

The bandwidth figures are vendor specs, not measurements I took. Treat the 6.6x ratio as approximate — nothing below depends on it being exact.

The model is Qwen3.8-27B, a hybrid-attention 27B: 64 layers in a fixed 3:1 ratio of gated-DeltaNet linear attention to full GQA attention, 262,144 native context extensible to ~1M via YaRN, and a multi-token-prediction head trained into the checkpoint so it can speculate against itself with no separate draft model. On the 5090 I served the NVFP4 build, RadixArk/Qwen3.8-27B-NVFP4; on the Spark I had run BF16 and FP8 a little earlier.

Both runs used SGLang’s own single-batch benchmark, batch size 1, 1,024 input tokens, 256 output tokens. Single stream — this is the “one person typing” number, not aggregate server throughput.

The numbers

Machine	Weights	MTP	Decode tok/s	TTFT	Prefill tok/s	Accepted len
DGX Spark	BF16	off	4.57	0.86 s	1,192	—
DGX Spark	FP8	off	7.80	0.53 s	1,927	—
DGX Spark	FP8	on	13.50	0.53 s	1,923	1.91
RTX 5090	NVFP4	off	54.64	0.09 s	11,124	—
RTX 5090	NVFP4	on	93.49	0.10 s	10,589	1.91

Prefill is 9x faster and time-to-first-token drops from 0.86 s to 0.09 s. The best configuration on the Spark (13.50 tok/s) loses to the worst configuration on the 5090 (54.64 tok/s) by 4x.

This is not a story about one machine being good and the other bad. They are built for different jobs, and the Spark’s job is to hold models the 5090 cannot load at all. But if the model fits in 32 GB after quantisation, the capacity advantage buys nothing and the bandwidth deficit costs everything.

Where the ceiling went

At batch 1, every decode step streams the whole weight set through memory once. That gives a hard ceiling: bandwidth divided by resident weights.

	Resident weights	Bandwidth roof	Measured	Fraction of roof
DGX Spark, BF16	~54 GB	5.1 tok/s	4.57	90%
DGX Spark, FP8	~27 GB	10.1 tok/s	7.80	77%
RTX 5090, NVFP4	20.1 GB	89.0 tok/s	54.64	61%

The Spark runs at 90% of its theoretical maximum. The 5090 runs at 61% of its own. Six times the bandwidth did not buy six times the throughput — it bought about 12x over BF16 and 7x over FP8, and left a third of the roof on the floor.

Correction, 15 August 2026. The paragraphs that followed here argued that the missing 39% was fixed per-step overhead, and generalised that to “the faster the machine, the more overhead dominates.” That was wrong, and the error was mine, not the hardware’s. The 61% figure measured a suboptimal software stack. On SGLang 0.5.17 with a pure-NVFP4 checkpoint, the same GPU and the same workload reach 87% of roof with per-step overhead of 1.57 ms (13%), not 7.06 ms (39%) — which puts the 5090 roughly where the Spark is, and removes the contrast the section was built on. I have left the original table above and replaced the analysis below. See What the stack was hiding, below, for the measurements and what actually caused it.

What the corrected numbers show is duller and more useful: at batch 1 a dense model is bandwidth-bound on any machine that is running a competent stack. Both boxes land near their roof once nothing is in the way.

	Measured step	Bandwidth-optimal step	Fixed overhead
DGX Spark, BF16	218.8 ms	197.8 ms	~21 ms (10%)
RTX 5090, NVFP4 (as first measured)	18.3 ms	11.2 ms	~7 ms (39%)
RTX 5090, NVFP4 (corrected stack)	12.1 ms	10.5 ms	~1.6 ms (13%)

Absolute overhead is far smaller on the 5090, and on the corrected stack it is a small share of the step as well. The 39% in the middle row was engine and checkpoint waste that a version bump and a better-quantised checkpoint removed.

The transferable finding survives in a weaker form. “This workload is memory-bandwidth-bound” is still a claim about a configuration and not just a model — but the thing that broke it in my first measurement was fixable software, not the arrival of a faster GPU. Before concluding that overhead now dominates your workload, check that you are not simply measuring a stack that has slack in it. I wasn’t, and I published the wrong generalisation for a day.

The Spark is built for a model shape this isn’t

It would be easy to read the table above as “the 5090 wins,” and that is the wrong conclusion. I picked a workload the Spark is architecturally unsuited to.

A dense 27B has to stream every parameter through memory on every decode step. There is no sparsity to exploit, so the bandwidth number is the whole story and the capacity number is irrelevant the moment the model fits. That is the worst possible shape for a machine whose distinguishing feature is a lot of slow memory.

Flip to a mixture-of-experts and both numbers change roles. An MoE in the 35B-A3B shape activates roughly 3B parameters per token while needing all 35B resident — call it 2 GB moving per step against 20 GB held. Capacity becomes the binding constraint and bandwidth mostly stops mattering, because you are no longer reading the model, you are reading a tenth of it. Run the same roof arithmetic and 273 GB/s supports something in the low hundreds of tok/s for that shape, against 13.6 tok/s for the dense 27B at the same precision. The machine that looks hopeless in my table looks entirely reasonable one model architecture over — and it can hold MoE checkpoints at precisions the 5090 cannot load at all, which is the actual point of buying a Spark.

This is reasoning, not a measurement. I did not benchmark an MoE on the Spark, and the arithmetic above ignores routing overhead, expert-gather patterns, and the fact that active-parameter counts understate real traffic. Treat it as the reason my comparison is unflattering by construction rather than as a number.

So: capacity gates which models you can run at all, bandwidth decides how fast dense ones go, and sparsity decides whether the bandwidth number matters. Buying a Spark to run dense models is the mismatch. It is not a bad machine; I gave it the one job its design specifically trades away.

MTP crosses the roof

Multi-token prediction drafts ahead with the model’s own head and verifies several tokens in one forward pass, so one weight read yields more than one token. It is the one optimisation that can beat the single-token bandwidth roof, and it does:

93.49 tok/s against an 89.0 tok/s ceiling. Above the line, which is only possible because the line assumes one token per weight read.

Two things transferred across the machines and one didn’t.

The acceptance rate transferred exactly. 1.91 accepted tokens per verification step out of 2 proposed, on both machines, to two decimal places. That is a property of the checkpoint and the workload, and it moved from one architecture to the other unchanged.

The speedup transferred. 1.73x on the Spark (7.80 → 13.50), 1.71x on the 5090 (54.64 → 93.49). Both machines get essentially the same multiplier from the same accepted length, which is what you would expect if both are bandwidth-bound and MTP is buying tokens per weight read. (I originally explained this by saying MTP also amortises the fixed per-step overhead “which is exactly what dominates here” — that rested on the 39% overhead figure I have since corrected. The simpler explanation is the right one.)

The cost did not transfer. The MTP head loads unquantised at 5.53 GB — a quarter of the 20.14 GB target model, for one extra layer — and on a 32 GB card that comes straight out of the KV pool:

	KV pool	SSM state	Free after
MTP off	36,685 tokens	4.78 GB	3.86 GB
MTP on	5,990 tokens	1.69 GB	6.52 GB

An 84% smaller KV pool. On the Spark this trade did not exist; here it is the whole decision. 93 tok/s with a 6k working set is not obviously better than 55 tok/s with a 37k one, and which you want depends entirely on what you do with the machine. I have not decided yet.

Note also that the recurrent SSM state, not the KV cache, is the expensive thing: 4.78 GB of fixed-size linear-attention state against 1.12 GB of KV for 36,685 tokens. The hybrid architecture moves the memory cost off the axis you are used to watching.

What the stack was hiding

Added 15 August 2026, after SGLang published 206.1 tok/s for this model on this GPU — 2.2x what I measured. They were right and I was slow. Chasing the gap found three separate causes, none of them hardware, and it is the most useful thing in this post.

Change	Decode tok/s	Gain
As published: SGLang 0.5.15.post1, mixed-precision NVFP4	54.64	—
Upgrade engine to SGLang 0.5.17	71.01	+30%
Swap to a pure-NVFP4 checkpoint	82.93	+17%
Add NEXTN MTP (accepted length 1.93)	118.17	+43%
Replace NEXTN with DSpark (accepted length 3.01)	159.14	+35%

Non-speculative decode went from 54.64 to 82.93 tok/s — 52% faster with no change to the GPU, the model, or the benchmark. That is what the 61%-of-roof number was really measuring.

1. The engine version was worth 30%. Identical flags, identical checkpoint, identical workload; only the SGLang version differed. I had no way to know this without running it, which is the argument for running it.

2. The checkpoint was worth 17%, and I got this one badly wrong. Asked to guess before measuring, I said the checkpoint was worth “maybe 1.06x” — reasoning from file size, since the build I used (20.42 GB) was already smaller than most alternatives. But size is the wrong variable. My checkpoint loads as MIXED_PRECISION: only 42% of it is actually 4-bit, with 38% at FP8 and 20% at BF16, averaging ~5.9 bits per parameter. The replacement loads as pure NVFP4 at 18.80 GB resident. A 7% reduction in bytes bought a 17% speedup, so this is not only about moving fewer of them — the pure-NVFP4 path uses different kernels. What is quantised matters more than how much.

3. The speculative decoder was the rest. My MTP configuration came from notes written for the other machine:

--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2

Two draft tokens per verification step. That is mathematically capped at 2x regardless of how good the model is, and I measured 1.91 accepted — 95.5% of the maximum that config permits. I had been reading that as a strong result. It was a strong result against a ceiling I had set myself, and I never questioned the ceiling because the number looked healthy.

DSpark drafts a block of 7 tokens per forward pass with a small trained draft model and a confidence head that sizes each verification window, giving a verify width of 8 against my 2. Accepted length went 1.93 → 3.01 and throughput 118.17 → 159.14 tok/s.

Two incompatibilities worth knowing about

The DSpark draft has no output head of its own. It ships 62 tensors — five attention layers, a projector, a norm, a confidence head — and borrows the target model’s lm_head. My original checkpoint quantised that head to packed 4-bit, which surfaced as:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x5120 and 2560x248320)

2560 is exactly 5120/2: two 4-bit values per byte. The draft was feeding a BF16 hidden state into a packed FP4 matrix. NEXTN worked on the same checkpoint only because its quantization_config happens to list mtp* in the ignore set. The fix is a checkpoint whose ignore set also includes lm_head — worth checking before you plan around any draft-model-based speculation.

Speculation costs memory that a 32 GB card does not have spare. The unquantised BF16 draft is 2.71 GB on top of an 18.80 GB target. At mem-fraction-static 0.96 the server OOM’d with 20 MiB free; 0.90 left enough headroom and is what produced the number above.

Where it still doesn’t reach

159.14 tok/s is 77% of SGLang’s 206.1. I did not close it further, and the honest list of remaining differences is: they presumably use their own NVFP4 build rather than a third party’s; their recipe specifies --attention-backend fa3 where I kept flashinfer so that the speculation algorithm stayed the only variable; and the accepted length depends on the workload — the draft’s own model card reports 3.39 averaged over eleven reasoning benchmarks against my 3.01 on a synthetic 1,024-token prompt. Any of those could be the rest. I stopped because the interesting question was answered.

The general lesson, which cost me a published mistake to learn: when your number is far off someone else’s on the same hardware, the gap is usually not one thing. Mine was three multiplicative ones, and the largest was a flag I had copied forward from a different machine without re-examining what it capped.

The context ceiling is free

I ran every configuration twice, at the native 262,144-token setting and at ~1M via YaRN:

Config	256k ctx	1M ctx
MTP off	54.64	54.60
MTP on	93.49	90.81

Identical within noise at 256k, and the 3% gap on the MTP row is a smaller KV pool (5,073 vs 5,990 tokens), not the rope scaling. Raising the configured ceiling changes what is reserved at startup, not what a 1,024-token prompt costs. This replicates what I saw on the Spark, and the hybrid architecture explains it: the SSM state is fixed-size regardless of context length, so most of the model does not care.

I did not test whether the 1M configuration produces good output at long context. I tested that it starts and that it does not slow down short prompts. Those are different claims and only the second one is mine.

Two things that were problems on one machine and not the other

Startup time stopped mattering. On the Spark, weight loading alone took 345 s and prefill CUDA-graph capture another 196 s, against a hardcoded 600 s startup timeout in the benchmark harness with no CLI override. Three of the four BF16 configurations died on that timeout — it is why the BF16 rows are missing from my table above and why I switched to launching the server separately and polling /health myself.

Here, weights load in 8.2 s and the server is healthy in 26–46 s. Same harness, same hardcoded 600 s, entirely invisible. I kept the separate-launch workaround because it was already written, but nothing on this machine needed it. A 42x faster weight load turns the single most disruptive problem of the first run into a non-event.

A documented config override crashes this architecture. The model card gives YaRN parameters to pass through SGLang’s --json-model-override-args. On this model that produces:

AttributeError: 'PreTrainedConfig' object has no attribute 'layers_block_type'

The override is applied as a flat config.update(...). For a model whose real settings live in a nested text_config, you have to pass that whole sub-config as a dict — and doing so replaces the engine’s own config subclass with a plain base config. The engine supplies layers_block_type (which layers are linear versus full attention) as a derived attribute on that subclass; it exists nowhere in the JSON. Overriding rope parameters silently deletes the layer map, and the model fails at layer construction, three abstractions from the thing I actually changed.

The fix is to not use the override at all. Write the modified config.json into a directory, symlink the weight shards next to it, and point the engine at that:

tc = config["text_config"]
tc["max_position_embeddings"] = 262144 * 4
tc["rope_parameters"] |= {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 262144,
}

Now the engine parses the file with its own config class and every derived attribute survives. Same result the override was supposed to produce, minus the crash.

The generalisation, which I have now hit twice in two months on two different runtimes: a config-override mechanism that merges dictionaries into typed objects is lossy at every boundary it crosses. It works fine for flat scalars and breaks silently for anything the object computes about itself.

Controls

I ran two of these twice, because the last time I benchmarked something I nearly published a 7% finding that was a slow sample.

The MTP comparison had a confound: the MTP server needed mem-fraction-static raised to 0.93 and the request cap dropped to 1 to start at all, so the 54.64 and 93.49 numbers came from differently-configured servers. Rerunning MTP-off with the MTP run’s exact memory settings gives 54.72 tok/s against the original 54.64 — a 0.15% difference. The memory tuning does nothing to single-stream decode and the MTP delta is clean.

I also reran both headline configurations unchanged, which is the cheapest experiment in any set:

Config	First run	Repeat	Spread
MTP off	54.64	54.64	0.00%
MTP on	93.49	93.44	0.05%

Run-to-run spread across full server restarts is under 0.1% on this box, tighter than the 1.5% I measured on a different card in June and tight enough that I trust three significant figures here. Every claim above clears that by two orders of magnitude, with one exception: the 3% gap on the MTP 1M row, which I have attributed to KV pool size and have not isolated.

I repeated the three corrected configurations too, on the same principle:

Config	First run	Repeat	Spread
0.5.17 + pure NVFP4, no spec	82.93	82.91	0.02%
0.5.17 + pure NVFP4, NEXTN	118.17	118.04	0.11%
0.5.17 + pure NVFP4, DSpark	159.14	158.89	0.16%

Accepted length reproduced exactly on both speculative runs (1.93 and 3.01). The spread is wider than the original set’s but still an order of magnitude below the smallest gain I attribute to anything.

What’s unproven

The bandwidth figures are spec sheets. I did not run a memory benchmark on either machine, so every “fraction of roof” number inherits whatever error is in 1,792 GB/s, and real achievable bandwidth is always lower than spec — which means the true fractions are higher than stated. That cut against me: it was another reason to distrust the original 61% before I published a generalisation on top of it.

The corrected figures were repeated once each and held to within 0.16%; see Controls. What I have not done is isolate the engine upgrade from the checkpoint swap at more than one point each, so the +30% and +17% are each a single A/B rather than a curve.

“Resident weights” is the engine’s reported memory usage after loading, not a measurement of bytes touched per decode step. It is close enough for a roof calculation and wrong in detail.

I did not compare NVFP4 against FP8 on the same card, because 27 GB of FP8 weights plus state does not fit in 31.4 GB. So “4-bit on the 5090 beats 8-bit on the Spark” conflates two variables and I cannot separate them with the hardware I have.

Most importantly: I measured speed and not one thing about quality. NVFP4 is a more aggressive quantisation than the FP8 the Spark ran, and 12x faster output that is meaningfully worse is not a win. I have no evaluation, no perplexity comparison, and no side-by-side against the BF16 original. Every number in this post is a throughput number and none of them says the model is still good.

That is the next thing to run. This one establishes only that the 5090 is much faster at producing tokens — and, after the correction, that the number you get from a given piece of hardware is a claim about your software stack until you have proved otherwise. I published that lesson the expensive way round.