Perplexity's Lily on an M5 Pro: Half the GPU, Half the Prefill, Twelve Minutes at 256K

Perplexity’s Lily is a small Metal inference server that does one thing: run Qwen3.6-35B-A3B at MLX 4-bit, batch 1, greedy, on an M5-generation Apple GPU. Their published report puts it at 194.7 tok/s decode on a 40-core M5 Max with 128 GB, roughly 1.3x MLX on the same machine.

I wanted the number for a smaller machine: an M5 Pro with a 20-core GPU and 48 GB. Same checkpoint revision, same lily-bench harness, same synthetic prompt, 64 decode steps, context from 4K up to the full 256K.

Prefill on the Pro runs at 0.48–0.49x the Max through 64K, which is the core ratio almost exactly. Decode holds up better than that at short context, 0.61x at 4K, and slides back to 0.48x by 128K. The whole 256K context fits in 48 GB with plenty to spare. What actually limits the machine is time: a 262,079-token prompt took 11 minutes 46 seconds to prefill, after which decode ran at 25.2 tok/s.

The setup

Lily is narrow on purpose. It validates the exact 35B-A3B architecture and quantisation layout at load time and refuses anything else, rejects sampling parameters and streaming, and requires Apple GPU family 10, which today means M5 or later. The Metal kernels are written for this one model shape.

The model is a hybrid MoE: 40 layers, of which 30 are gated-DeltaNet linear attention and 10 are full attention, with 256 experts and 8 active per token. The checkpoint is 19 GB on disk.

The harness builds a deterministic pseudo-random prompt (token i is a multiplicative hash of i, mod the vocabulary size), runs an exact-shape warmup, then times one prefill and 64 decode steps in a fresh process. Prefill is TTFT-style: the timer stops when the first generated token has been read back. I ran the harness directly, one pass per context, not through the repo’s matrix runner.

Machine	GPU cores	Memory
M5 Max (Perplexity’s report)	40	128 GB
M5 Pro (mine)	20	48 GB

The full sweep

Prompt tokens	Prefill time	Prefill tok/s	Decode tok/s	Decode ms/token
4,096	1.5 s	2,789.6	112.1	8.92
16,384	6.8 s	2,420.2	96.4	10.37
65,536	49.6 s	1,321.7	68.5	14.59
131,072	3 min 14 s	674.7	44.2	22.65
262,079	11 min 46 s	371.1	25.2	39.65

The last row is 262,079 rather than 262,144 tokens so that the prompt, the 64 decoded tokens and one lookahead token exactly fill Lily’s 262,144-token limit.

Against the M5 Max

Four of my context lengths overlap with Perplexity’s published table.

Context	Pro decode	Max decode	Pro / Max	Pro prefill	Max prefill	Pro / Max
4,096	112.1	182.8	0.61x	2,789.6	5,724.4	0.49x
16,384	96.4	167.0	0.58x	2,420.2	4,968.5	0.49x
65,536	68.5	125.4	0.55x	1,321.7	2,752.5	0.48x
131,072	44.2	92.7	0.48x	674.7	1,792.7	0.38x

Prefill is almost exactly half, up to 64K. Lily prefills in 4,096-token chunks with batched matrix multiplies, which should be mostly compute-bound, and half the GPU cores gives 0.48–0.49x the throughput. That’s as clean a scaling result as I’ve had between two different machines. At 128K it drops to 0.38x, so the Pro falls further behind once attention dominates the prefill. I don’t know why it degrades more there than the Max does.

Decode doesn’t track core count at short context. At 4K the Pro gets 61% of the Max’s decode with 50% of its cores. A batch-1 decode step mostly streams the active expert weights rather than computing on them, so I’d expect it to depend more on memory bandwidth and fixed per-step overhead than on how many cores there are.

Splitting the step time into a fixed part and a part that grows with context makes this visible. Here’s a straight-line fit of ms/token against context over the four shared points:

	Fixed (intercept)	Per 1K tokens of context
M5 Max	5.3 ms	0.042 ms
M5 Pro	8.4 ms	0.106 ms
Pro / Max	1.6x	2.5x

The fixed part (expert weights, routing, and the linear-attention layers, whose state doesn’t grow) costs the Pro 1.6x. The part that grows with context costs it 2.5x, which is worse than the core ratio. Ten of the forty layers are full attention, and every decode step reads their whole KV cache. At short context that read is small and the Pro looks good. By 128K it’s more than half of the Pro’s step time, and the decode ratio has fallen to 0.48x.

Some arithmetic, which is reasoning and not a measurement: Lily stores K and V as bf16 in [2 heads, seq, 256] buffers per full-attention layer, so ten layers hold 20 KB per token. If the KV read were the only thing that grew with context, the slopes above would imply roughly 190 GB/s of effective attention read on the Pro and 490 GB/s on the Max. I haven’t measured either machine’s bandwidth, and attention kernels do more than stream bytes, so treat those figures as rough magnitudes.

Memory isn’t the limit at 48 GB

The weights are 19 GB. At the full 262,144-token capacity the KV cache is 5.4 GB in bf16, because only a quarter of the layers keep one. The 30 linear-attention layers carry a fixed-size state instead. That fits comfortably in 48 GB alongside everything else on the machine. I didn’t measure peak resident memory; the figure comes from the model config and the cache layout in the source, not from a reading.

So the M5 Pro can hold the whole context window. The real question is whether you’d want to wait for it.

The wait

Prefill time stops scaling linearly early. Going from 16K to 64K, four times the context took 7.3x the time. Each doubling after that took 3.9x (64K to 128K) and then 3.6x (128K to 256K). Attention cost is quadratic in prompt length, and by 64K it dominates.

In practice, a 16K prompt prefills in under 7 seconds and a 64K prompt in 50. A 128K prompt takes three and a quarter minutes; Perplexity’s throughput figure for the Max implies about 73 seconds for the same prompt. A 256K prompt takes nearly twelve minutes. Decode at the far end is still usable at 25 tok/s, so the machine can work with a full context. The cost is getting there.

That makes Lily’s session cache matter more on this machine. The server keeps a two-entry LRU of decode states and reuses one whenever its tokens are an exact prefix of the next prompt, so in a growing conversation you pay the full prefill once and then only for each new turn. I didn’t benchmark that path; every number above is a cold prefill. But it’s the difference between a three-minute wait once and a three-minute wait on every turn.

Something I can’t explain

At short context, decode step times are flat. At 4K, the 64 steps have a standard deviation of 0.09 ms around 8.90 ms, and the first sixteen and last sixteen steps average within 0.04 ms of each other.

At long context they aren’t flat. At 128K the mean step falls from 23.7 ms over the first sixteen steps to 21.9 ms over the last sixteen, an 8% drop. At 262K it falls from 41.0 ms to 35.3 ms, a 14% drop, with a standard deviation of 3.3 ms, and the first step took 47.0 ms.

The 25.2 tok/s reported at 262K is the average over all 64 steps. If the downward trend continued, a longer generation would run faster than that. I don’t know whether this comes from GPU clock behaviour after a twelve-minute prefill, something in the cache, or something else, and one pass isn’t enough to tell. The post-run check reported no thermal or performance warning.

What’s unproven

Each context length was measured once. Perplexity’s numbers are the median of two rounds through a matrix runner that enforces cooldowns between arms, refuses to run on battery, and checks that each engine’s 64-token output repeats exactly across rounds. Mine are single passes run directly. I have no run-to-run spread for this machine, and nothing in my result files records the power source. Every ratio above rests on one sample per context.

I didn’t run the MLX baseline on the Pro. Perplexity’s report shows Lily at about 1.3x MLX on decode on the Max. Nothing here tells you whether that holds on the Pro, and I’m not claiming it does.

The two machines differ in more than the GPU. Their report was taken on macOS 27.0 and mine on macOS 26. My result files record the source identity as unknown because I built without setting it, so I can’t tie these numbers to an exact Lily commit.

The prompts are synthetic hashes, so the output is meaningless. At 128K the 64 generated tokens contain two distinct IDs, and at 262K, five. Different tokens route to different experts, so the Pro and the Max didn’t do identical work, a caveat Perplexity’s own report makes about its engine-to-engine comparison.

And this is a throughput post. Nothing here says anything about the quality of what a 4-bit 35B MoE produces with 256K of context. I tested that it runs at 256K, not that it can make use of 256K.