My 256K Benchmark Was Fast Because It Wasn't Cold

I was testing Inco’s Splash 1.3.2 on a 48 GB M5 Pro against an earlier Lily context sweep. My first 256K result said Splash prefilled a 262,079-token prompt at 906.7 tokens per second. Lily, using the same target checkpoint, had recorded 371.1. That made Splash look 2.44x faster.

The result was wrong. Splash had reused 131,040 tokens from an earlier request.

With the cache genuinely cold, the same prompt took 418.1 seconds instead of 289.0 and prefill fell to 626.9 tok/s. Splash was still faster in this test, but by 1.69x rather than 2.44x. The interesting result was not which engine won. It was how easily a deterministic benchmark turned into a cache benchmark without telling the harness.

The test setup

The machine was a MacBook Pro with an Apple M5 Pro, 20 GPU cores and 48 GB of unified memory, running macOS 26.5.1. I served mlx-community/Qwen3.6-35B-A3B-4bit through Splash 1.3.2, pinned to the same checkpoint revision as the Lily results. Splash paired the target with its matching DFlash2 draft model and exposed an OpenAI-compatible API on the Mac’s loopback interface. The server supported the model’s full 262,144-token context window.

I kept Splash running as a per-user macOS service and drove it with a separate Python harness. Model loading was outside the timer. The harness did not send chat messages or apply a chat template. It called the completions endpoint with raw integer token IDs so I could reproduce Lily’s synthetic context-length workload as closely as the two APIs allowed.

For each context length, the harness:

1. Built a deterministic pseudo-random prompt of 4K, 16K, 64K, 128K or 262,079 tokens.
2. Requested 65 output tokens with greedy decoding, end-of-sequence stopping disabled and HTTP streaming enabled.
3. Measured TTFT from immediately before the request to the first non-empty streamed token.
4. Measured decode across the 64 intervals between the first and sixty-fifth output token.
5. Replayed the identical request immediately to measure cached TTFT separately.

Splash prefill throughput was prompt tokens / TTFT, so its number includes request serialization, the local HTTP hop and server scheduling. Lily’s saved benchmark ran the same target checkpoint in-process: one measured prefill after an exact-shape warmup in a fresh process, followed by 64 target-only decode steps. Neither side includes model load time.

The prompts were synthetic and the 65 generated tokens were not evaluated for quality. This was a serving-speed test, not a model-quality test.

How I accidentally warmed the prompt

The benchmark builds synthetic prompts from a multiplicative hash of each token position. That makes runs reproducible, but it also means every longer prompt begins with the entire shorter prompt.

I knew that was dangerous, so I changed the first token of each prompt. My first process used sentinel values 1, 2 and 3 for the 4K, 16K and 64K tests. Then I ran the 128K and 256K points in separate invocations.

The counter restarted at 1 each time.

The 128K request therefore began with the same sentinel and deterministic suffix as the earlier 4K request. The server reported 4,064 cached tokens. The 256K request then matched the 128K request and reported 131,040 cached tokens. A new benchmark process did not imply a new server cache.

The client only measured the time from sending the request to receiving the first token. It had no idea how much of the prompt the server actually processed. The arithmetic was internally consistent:

prefill throughput = prompt tokens / time to first token

The numerator was the bug. I counted all 262,079 prompt tokens even though the server had already materialised half of them.

The log was the ground truth

The suspicious result was not obvious from the HTTP response. It was obvious in the server log:

Started · input 262,079 · cached 131,040
Done · input 262,079 · output 65 · TTFT 289.0s

I changed the harness to choose a fresh first-token salt for every invocation and record that token alongside the result. Then I reran the long-context points and checked the log before accepting them:

Started · input 262,079 · cached 0
Done · input 262,079 · output 65 · TTFT 418.0s

The relevant part of the fix is small:

run_salt = time.time_ns()

for index, context in enumerate(contexts, start=1):
    first_token = (run_salt + index) % tokenizer_size
    prompt = synthetic_prompt(context)
    prompt[0] = first_token

The timestamp is not there for randomness in the statistical sense. It is a cache-busting nonce. Recording it keeps the generated prompt reproducible enough to audit later.

The corrected sweep

These are the cold-prefill results after verifying cached 0 for every Splash row:

Context	Splash prefill	Splash TTFT	Lily prefill	Lily TTFT	Splash / Lily
4,096	3,298.2 tok/s	1.2 s	2,789.6 tok/s	1.5 s	1.18x
16,384	3,020.0 tok/s	5.4 s	2,420.2 tok/s	6.8 s	1.25x
65,536	1,827.4 tok/s	35.9 s	1,321.7 tok/s	49.6 s	1.38x
131,072	1,114.4 tok/s	117.6 s	674.7 tok/s	194.3 s	1.65x
262,079	626.9 tok/s	418.1 s	371.1 tok/s	706.2 s	1.69x

The real pattern survived the correction: Splash’s prefill advantage grew with context length. The headline ratio did not.

An exact replay makes the size of the trap clearer. Splash returned the first token from the cached 256K prompt in 229 ms, against 418 seconds cold. That is more than a 1,800x difference in TTFT. Prefix caching is supposed to be dramatic. It is what makes a growing long-context conversation practical. It just has to be named as part of the workload.

Decode is a different comparison

I generated 65 tokens per request and measured the 64 intervals after the first token. That mirrors Lily’s 64 decode steps, but the engines are not doing the same work.

Splash used a DFlash2 draft model and verified several proposed tokens per target pass. Lily used target-only, depth-2 pipelined decode. At 256K, Splash produced 83.3 tok/s while Lily produced 25.2. The Splash log reported 3.8 tokens per speculative cycle for that request.

That is a useful end-to-end serving result. It is not evidence that Splash’s target-model decode kernel is 3.3x faster. A different generated sequence can change draft acceptance and move the result sharply, which is why decode bounced between wins and losses across the sweep.

What I would require next time

For a cold-prefill benchmark, the client measurement is not enough. I now want four things for every row:

1. A unique prompt prefix, not merely a fresh client process.
2. A server-side cache counter proving that the cold run processed zero cached tokens.
3. The exact replay measured separately and labelled as cached.
4. The prompt identifier or salt stored with the raw result.

I would also keep cold and cached performance in the same report. The cached number is not cheating when it represents a real multi-turn session. It is cheating only when it is reported as cold.

What this still does not prove

This was one measured pass per context, not a distribution. I do not have confidence intervals.

The environments were not controlled. The Splash run saw memory-pressure warnings and a thermal state change from nominal to fair. The saved Lily run reported no warning and much more free memory. The hardware metadata also disagrees: the current machine reports a 20-core GPU while the older Lily report says 18 cores.

The synthetic prompts have the same shape and distribution but are not token-for-token identical. Lily samples from the checkpoint’s padded 248,320-row output matrix. Splash rejects the padding-only IDs and samples from the 248,077-token tokenizer instead.

So I would not use this sweep to claim a universal 1.69x engine advantage. I would use it to make a narrower claim: on this machine, in these single runs, Splash reached the first token sooner at every tested context after I proved the cache was cold.

The stronger lesson is about the failed run. If the server has a prefix cache, cache state is part of the benchmark whether the harness acknowledges it or not.