Notes on solutions architecture, LLM agent systems, and offensive security. Written in the open, mostly while the details are still fresh.
A 2-bit quantised 35B MoE runs at 126 tok/s on a 16 GB consumer GPU, within 1.5% of the published figure. The tuning recipe shipped alongside those numbers was worth 3% instead of 20%, and two of its knobs couldn't be set at all.
My first time serving a model on a rented per-second GPU instead of my own card. vLLM wouldn't load the GGUF file, the CUDA build had to link against stub libraries because image builds get no GPU attached, and the context flag I thought I understood was dividing my window across concurrent request slots.
Turning on multi-token prediction in llama.cpp took a 27B model from 58 to 136 tokens per second. The same upgrade that enabled it also broke my existing config by using more VRAM — the win and the regression shipped together.
Two retrieval pipelines over the same corpus, answered by the same model with the same prompt, overlapped on 1.8 of 12 retrieved documents — and failed in opposite, predictable ways.
Fifteen years of notes that lived in Slack DMs, call recaps, and scratch files — collected somewhere they can actually be found.