Escha W2 on an RTX 5060 Ti: The Benchmarks Reproduced, The Tuning Advice Didn't

The subjective part first: this is the best and fastest model I have run locally. It is what I now use for day-to-day assistant work and for driving browser agents, and it is the first model on this box that has been quick enough and capable enough to hold both jobs at once.

That is an impression, not a measurement, and I have kept it separate from everything below on purpose. What I benchmarked was speed. What made me switch to it was the other thing.

I spent an evening standing up a 2-bit quantised 35B mixture-of-experts model on a 16 GB consumer card — Qwen3.6-35B-A3B-Escha-W2, an Apache-2.0 release from Escha Labs, served through their SGLang-based runtime. The weights are 12.3 GB on disk: 35B parameters across 256 experts, about 3B active per token, compressed roughly 5.6x from the 16-bit original.

The published benchmark table for my exact card reproduced almost exactly. The tuning recipe printed two sections below it did not, and the gap between those two outcomes is the part worth writing down.

The setup, briefly

One RTX 5060 Ti, 16 GB, consumer Blackwell (sm_120). The runtime ships as a wheel plus a serve.sh launcher, in a separate repository from the weights. I wanted maximum decode speed at up to two concurrent slots — not a throughput server, just a fast local box for one person and the occasional second request.

The install has one real trap, and it is documented: PyTorch must be pinned to 2.9.x, because the compiled extension is ABI-linked against libtorch and the wheel declares no torch dependency at all. An unpinned torch>=2.9 resolves to 2.11 and you find out at import time, after two multi-gigabyte downloads. The second trap is transformers below 5.8, which does not fail — it logs one warning and then serves fluent nonsense. Both of these are called out clearly in the docs. I mention them because they are the two ways this stack breaks without telling you, and neither produces an error at the moment you make the mistake.

Final config, after tuning:

ATTN_BACKEND=triton   # mandatory on sm_120; the default backend asserts on hybrid-GDN
INT8=on               # int8-as-stored for lm_head/embed/attn/GDN/shared-expert
RADIX=0               # prefix caching off
GRAPHS=1  CUDA_GRAPH_BS="1 2"
MAXREQ=2  MAXMAMBA=2
MEM=0.92  CTXLEN=32768  CHUNK=2048

That lands at 15.3 GB of 16.3 GB used, an 88,293-token KV pool, and the model’s full 32k context window available to both slots.

The benchmarks reproduced

Single-stream decode, greedy, thinking disabled, 512 output tokens. My numbers against the vendor’s published table for the same card:

Prompt	Mine	Published	Delta
short (~10 tok)	126.4 tok/s	127.9	−1.2%
~1,000 tok	122.8 tok/s	124.0	−1.0%
~4,000 tok	114.7 tok/s	116.5	−1.5%

Time to first token was 0.26 s short, 0.82 s at 1k, 1.93 s at 4k. Two concurrent streams gave 103 tok/s each, about 203 tok/s aggregate.

Within 1.5% on every row, on a card I bought rather than one the vendor sampled. That is rarer than it should be, and it is worth saying plainly: the headline numbers were honest. Decode barely moves with prompt length here, which tracks — at batch 1 this is bandwidth-bound on the weights, and the KV read is small next to them.

I ran each figure three times per server start and took the best; run-to-run spread on single-stream was about 1.5%.

The tuning advice didn’t

The same documentation that produced those numbers ships a tuning section. Its most emphatic claim is that disabling the radix prefix cache is worth about 20% of single-stream decode, with a measured 187 → 225 tok/s to back it up.

On my card, at two slots, it was worth 3.4%.

Config (both MAXMAMBA=8, identical 69,429-token pool)	bs1 decode
RADIX=0	127.1, 126.8 tok/s
RADIX=1	122.8, 122.8 tok/s

The vendor’s number is not wrong. It was measured on a 24 GB card at batch 32, where the overlap scheduler that the radix cache disables has thirty-two requests’ worth of CPU work to hide behind GPU compute. At two slots there is almost nothing to overlap, so the mechanism that produces the 20% barely engages.

That is the transferable bit, and it generalises past this model: a benchmark number describes the hardware, but a tuning delta describes the hardware and the workload it was measured under. The first travels to your box. The second frequently doesn’t, and tuning guides rarely state the concurrency their percentages were measured at — this one did, and I still had to run the A/B to notice it didn’t apply to me.

I kept RADIX=0 anyway. Three percent is three percent, and I have no prefix reuse to trade it for.

Two knobs that turned out not to be knobs

INT8=off is not available on a 16 GB card. The docs present int8-as-stored as a concurrency tradeoff: on below the crossover, off above it, crossover somewhere between 4 and 8 concurrent streams. That reads like a decision. It isn’t one here — turning it off dequantises those layers to fp16 and wants about 2.2 GB I don’t have. The server refuses to start:

Not enough memory. Please try to increase --mem-fraction-static.
Current value: mem_fraction_static=0.92

The advice in that message is a dead end. 0.92 is already the documented ceiling for a 16 GB card — above it, an out-of-memory inside the fused MoE kernel takes the whole server down rather than failing one request. I tried 0.94 anyway to see whether the error’s own suggestion could work. Same failure. So the message points you at the one change the documentation specifically warns against, and that change doesn’t fix it either.

RADIX=1 at MAXMAMBA=2 doesn’t start at all. This one was my favourite. Effective concurrency on this hybrid-SSM model is clamped to min(MAXREQ, MAXMAMBA // ratio), where the ratio is about 4 with the radix cache on and about 1 with it off. My config had MAXMAMBA=2, so turning radix back on made the effective slot count 2 // 4 — zero. The server died with:

AssertionError: capture_bs=[0]

Nothing in that message mentions the radix cache, the mamba pool, or concurrency. It names the CUDA graph batch-size list, which is the thing that consumed the zero rather than the thing that produced it. If I hadn’t already read the clamp formula that afternoon I would have spent the next hour editing CUDA_GRAPH_BS.

An error naming a value is not an error naming a cause. Two knobs interacted, and the assertion fired in a third place downstream.

The finding I nearly published

While isolating the radix effect I changed MAXMAMBA from 2 to 8 and the two-stream aggregate went from 186.8 to 199.4 tok/s. Call it 7% for free.

I almost wrote that down. Instead I re-ran the original config, unchanged, and got 202.4 tok/s — higher than the “improvement.” The 186.8 was simply a slow sample.

Single-stream decode on this box is stable to about 1.5% across restarts. Two-stream aggregate swings about 8% for an identical config. I had been reading both off the same table as though they carried the same weight, and one of them could not support a 7% claim at all. MAXMAMBA=2 stayed, because at equal throughput it buys a 27% larger KV pool (88,293 tokens versus 69,429) and that is what makes the full 32k window fit on both slots.

Same lesson as the last time I benchmarked something and it bit me in a different place: run the control. The cheapest experiment in the set is the one where you change nothing.

What I couldn’t get

The biggest single speedup available to this architecture is multi-token prediction — the model drafts several tokens ahead with its own head and verifies them in one pass. I measured 2.3x from exactly that on a different model a couple of months ago.

The launcher exposes it, the checkpoint contains mtp.* tensors, and it cannot be used: the published release doesn’t include the separate draft export the runtime needs, and the head that ships in the weights isn’t servable as one. The vendor documents this. At two concurrent slots it would have been the largest win on the table by a wide margin — MTP helps most exactly where batching helps least — and it simply isn’t reachable from the files that ship.

What I’d generalise

• Reproduce the benchmark, re-derive the tuning. Published throughput figures survived the trip to my hardware; published tuning percentages didn’t survive the trip to my workload. Those are different kinds of claim and deserve different amounts of trust.
• Check whether a documented knob is actually settable on your box before you reason about which setting is better. Two of mine had exactly one legal value, and I’d written a justification for one of them before I tested that the alternative could even boot.
• Match the metric’s variance to the size of the claim. An 8%-noisy measurement cannot support a 7% finding. I got that wrong for about ten minutes.
• Read error messages as symptoms. capture_bs=[0] was three inferences away from its cause.

What’s unproven

I verified from the startup log that every knob took effect, and I A/B’d the two I made the strongest claims about. I did not A/B the rest. CUDA graphs are reportedly worth 4.4x on this architecture and I believe it, but GRAPHS=1 is the default and I never ran the eager control, so that number is the vendor’s and not mine.

I also measured speed, not quality. Beyond checking that arithmetic came back correct and the reasoning trace parsed into the right field, I ran no evaluation. So the claim I opened with — best local model I have used, the one I now point my assistant and browser agents at — is exactly that: a claim. It comes from using the thing daily, not from scoring it, and you should read it with the weight you would give any other person’s impression. The published quality case for this 2-bit build — near-parity with an FP8 baseline across six axes, 0.069 nats of KL divergence from BF16 — is entirely the vendor’s measurement, on the same page as the throughput table that did reproduce. That earns it some credit. It does not earn it a citation from me, and I have no idea yet whether the one real gap they report, long-horizon code generation, shows up in the work I actually do on this box.

That is the next experiment. This one only establishes that the thing runs fast.