First Experiments With Serverless GPUs and the 128k Context I Was Advertising Was Really 16k Each

I run a 27B Qwen3.6 community finetune locally on a 32 GB card, which forces a quantisation and context-length compromise I don’t love. So I tried the obvious upgrade: rent a 96 GB GPU by the second, serve the higher-precision Q8_0 file from it, and keep the 128k context window the model supports.

This was my first real go at serverless GPU hosting — Modal specifically, though most of what bit me is not Modal-specific. The pitch is that you write a Python file describing a container and a GPU, deploy it, and get an HTTP endpoint that scales to zero when nobody is using it. That part was genuinely as easy as advertised. The things that cost me the day were all downstream of one structural difference from running on my own machine: the environment that builds your image is not the environment that runs it. Locally those are the same computer, and every assumption I didn’t know I was making came from that.

It works, and the finished config does 42.67 tokens/sec of generation. Three things surprised me on the way there, and only one of them was about the GPU.

vLLM couldn’t load the file, for a reason that isn’t about vLLM

First attempt was vLLM, because it’s the default answer for serving a model to more than one caller at a time. It has an experimental GGUF loader. It refused the file:

ValueError: GGUF model with architecture qwen35 is not supported yet.

The instinct is to go read vLLM’s supported-models list, find Qwen3, and be confused — because Qwen3 is supported. But that error isn’t coming from vLLM’s model registry. vLLM’s GGUF path delegates checkpoint parsing to Hugging Face transformers, so the question isn’t “does my server support this architecture,” it’s “does the library my server hands the file to support this architecture.” Two different lists, and the second one is invisible from the documentation of the first.

The generalisable version: when a loader is described as experimental, find out what it delegates to before you debug it as if it were native. Architecture support for a format and architecture support for a model family are separate things, and a community finetune of a recent model tends to sit exactly in the gap. I stopped fighting it and moved to llama-server from llama.cpp, which reads GGUF as its native format rather than as an import path.

The image builder links CUDA it can’t see

llama-server with CUDA support means building it, and building it into the container image so it isn’t recompiled on every cold start. The GPU is attached when the container runs, not when the image is built — that’s how every serverless GPU platform works, and it’s correct, since you shouldn’t be billed for an H100 to run cmake. But it means the build step has no /dev/nvidia*, no driver, and therefore no real libcuda.so.1 to link against:

/usr/bin/ld: warning: libcuda.so.1, needed by libggml-cuda.so, not found

This one has a clean answer that I always forget exists. CUDA ships stub libraries in /usr/local/cuda/lib64/stubs — symbol-complete, functionally empty. They exist so you can link at build time on a machine that will never run the binary. Point the linker at them and the build completes:

LIBRARY_PATH=/usr/local/cuda/lib64/stubs
-DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda"
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda"

At runtime the real driver library is present and takes precedence, so the stub is a build-time fiction only. Do not copy the stub directory into the runtime library path “to be safe” — a binary that resolves libcuda to the stub will start cleanly and then fail in ways that look like hardware problems.

Two other build flags mattered more than I expected:

• -DGGML_CUDA_FA_ALL_QUANTS=OFF — the default compiles a FlashAttention kernel for every combination of KV cache quantisation types. Turning it off took the CUDA build from roughly five minutes to about twenty seconds. I use one KV type; I was paying for dozens.
• -DCMAKE_CUDA_ARCHITECTURES="89;90" — naming the two architectures I actually target instead of letting CMake fan out across every generation it knows.

Both are the same lesson, which is that an image build you run once still costs you every time you change the line above it. A four-and-a-half-minute compile is a four-and-a-half minute wait on every iteration of the surrounding config, and that’s where the whole day goes.

The flag I had wrong

Here is the part worth the post.

I had the server started with a 128k context window and eight parallel request slots, and I wrote that up as “128k context, 8 concurrent users.” Both flags were doing what the documentation says. What the documentation says is not what I had assumed:

-c 131072      # total KV cache budget
-np 8          # split across this many slots

-c is not per-request context. It’s the size of the whole KV cache, and -np divides it. 131072 / 8 = 16,384 tokens per user. I had built a 128k-context server and then partitioned it into eight sessions with a 16k window each — smaller than the 32 GB workstation config I was trying to improve on. Nothing errored. Each slot just silently has one-eighth of the number in the flag I was quoting.

I dropped to -np 4, which gives four concurrent sessions of 32k each, and that’s the real trade: concurrency and context length are the same budget, and you can’t quote both numbers at full value in the same sentence. If you serve long-context requests, you serve few of them at once. Pick which one the workload actually needs before you size the GPU, because the GPU is the thing you’re paying for and it can’t tell you which way you got this wrong.

The VRAM split shows where the money goes. Of about 87 GB used on a 96 GB card:

Component	Approx VRAM
Q8_0 weights, 27B params	28.5 GB
KV cache, 131,072 tokens at f16	56.0 GB
CUDA context and overhead	2.5 GB

The cache is twice the model. That ratio is why the context/concurrency question is the sizing decision and the weights are almost a footnote — and it’s an argument for testing quantised KV before buying a bigger card, which I haven’t done yet.

What I left on the table

The honest number: 42.67 tokens/sec generation, 145.35 tokens/sec prompt evaluation, from the server’s own telemetry.

My 32 GB workstation does 136 tokens/sec on the same model family. The rented 96 GB card is three times slower, and I know why — the multi-token prediction head I benchmarked last week is present in this GGUF file and I never passed the flag that uses it. I picked the MTP build of the model and then ran it as an ordinary one.

So treat 42.67 as a floor rather than a result. It’s a correct config, not a tuned one, and the difference between those two is a flag I already knew about.