One command rents the GPU and joins it to a tailnet; one gives it back. In between: +82% from a draft head that ships inside the checkpoint — and a 3.6× cut to concurrency that no throughput table shows you.
At concurrency 1 it is worth +82%. At concurrency 64 it loses,
4,242 against 5,694 tok/s, because it has taken the card from 175 concurrent slots down to 48
and the surplus requests queue. Both numbers are real and they point opposite ways. Pick the
mode for your workload — SPEC_K=0 reverses it — rather than inheriting a default
and a benchmark that disagree.
One pod, one checkpoint, one flag set. SPEC_K is the number of draft
steps; SGLang wants num-draft-tokens = steps + 1, because it counts the
always-committed target token. Everything else is held.
k=4 is the point of the chart. An optimum you have not seen stop is just the last point you measured. Running one budget past the peak is what turns "k=3 was the best we tried" into "k=3 is the optimum, and here is why it stops".
Acceptance length rises the whole way — 1.975, 2.95, 3.875, 4.625. Read only that and k=4 looks like the winner. The per-token acceptance rate is what actually turns over.
accept_length includes the always-committed target token, so
its floor is 1. vLLM's does not, so its floor is 0. They differ by exactly one. The rate
plotted here divides accepted draft tokens by the budget, which removes that offset —
but establish the floor before comparing any published acceptance figure to this one.Once per-token acceptance starts to fall, each further draft position costs more verify compute than it returns. That is the whole story, and it is invisible in the length column.
Every speculative arm beats the baseline at low concurrency and loses at high concurrency. Not because drafting gets slower — because speculation took the card's concurrency slots.
The crossover above has a cause, and the engine prints it at startup. Turning
speculation on takes the KV pool from 2.34 million tokens to about half a million, and
max_running_requests from 175 to 48 — leaving roughly 80 GB of a 178 GB card
unclaimed.
--max-mamba-cache-size 900 to reclaim the idle memory made it worse: KV fell
506k → 287k with concurrency unchanged.The pod is reachable only over a tailnet, which is the point of the design. It is also the fastest way to publish a completely wrong number.
benchmark.sh therefore copies the probe to the pod and runs it against
127.0.0.1; LOCAL=1 opts out and stamps the output
DO NOT QUOTE AGGREGATE.Lazy CUDA-graph and kernel warmup — 24.1 against 88.8 tok/s on the same server. N=1 is the column people quote, so an unwarmed run publishes a warmup measurement wearing a throughput label. Every probe here sends a warmup request first.
lmsysorg/sglang:latest moves
Every number on this page is sglang 0.5.17. A pod rented later is a different engine, and on a related host an engine-layer bump alone moved prose throughput ~9%. Pin the image to a digest, or compare only against your own baseline.
A B200 can hold Qwen3.8-27B at BF16 and a usable KV pool, which makes
the control cheap: one pod, one flag set, two --model-path values. Most boxes
cannot run this comparison at all.
| arm | N=1 | N=8 | N=16 | N=32 | N=64 | weights | KV pool | slots |
|---|---|---|---|---|---|---|---|---|
| BF16 | 93.6 | 606.5 | 1,279.2 | 1,961.8 | 3,333.0 | 51.05 GB | 1,850,304 | 138 |
| NVFP4 | 152.5 | 1,038.1 | 1,922.8 | 3,269.8 | 5,306.1 | 22.67 GB | 2,339,968 | 175 |
| ratio | 1.63× | 1.71× | 1.50× | 1.67× | 1.59× | 0.44× | 1.26× | 1.27× |
The missing 0.6× is the state cache. Decode reads 51.05 GB of BF16 weights against 22.67 GB quantized, which alone predicts 2.25×. It does not arrive, because the hybrid GDN state work is not quantized: the mamba cache is 50.78 GB on BF16 and 64.21 GB quantized, and that traffic is BF16 in both arms. The ratio is flat across concurrency, so this is not a batch effect — roughly a third of decode traffic is untouched by the weight format, which caps the return on going lower-precision on this architecture.
| B200 | RTX PRO 6000 Blackwell SE | |
|---|---|---|
| price | $6.79/hr | $2.09/hr |
| VRAM | 178 GiB | 96 GiB |
| compute capability | SM 10.0 | SM 12.0 |
| attention backend | trtllm_mha | flashinfer |
| KV pool | 2,339,968 | 1,016,563 |
| max_running_requests | 175 | 76 |
| N=1 | 152.5 | 42.9 |
| N=64 aggregate | 5,306 | 2,121 |
| cost per 1k output tok at N=64 | $0.00036 | $0.00027 |
Rent the B200 when latency or concurrency binds. Rent the cheaper part when total spend
binds. GPU_TYPES pins either — and the capacity fallback will hand you the cheap
one anyway on a dry day, still named b200. Read the hourly rate in
./status.sh before assuming which you got.
976 requests over 1.64 h at 34.9% duty cycle — under the 78.6% break-even, so the arithmetic says serverless wins. It does not. The mean gap between requests is 4.1 seconds; no flex worker scales to zero on that, so it bills continuously at a rate 27% above the pod. Serverless needs idle stretches longer than the ~3-minute cold start.
The bigger lever was never the billing model. That
same workload ran at concurrency 1 on a card sized for 175, yielding 39 output tok/s averaged
over wall-clock from hardware that does 5,306. Pushing the client to 8–16 in flight finishes the
queue several times sooner at the same hourly rate — and saves far more than the $1.85/hr
serverless was ever going to. Note the interaction with everything above: at
SPEC_K=3 the ceiling is 48, not 175.
The default checkpoint is Qwen's own FP8. Three of the four reasons for that are evidence. The fourth — the one people will want to quote — is not.
With speculation on, ~78 GB of the card is already unclaimed and FP8 and NVFP4 land at the same 48 slots. The usual argument for 4-bit does not apply on 178 GiB.
Builds carrying it differ by up to 52% on prose because they disagree
about which modules they quantize. One publishes a recipe that does not reproduce its own
shipped config. Read quantization_config, always.
mtp is in its ignore list. Checkpoints that omit it make the
loader die on a fused name present in neither targets nor ignore.
The bootstrap detects and works around that, but it is one more thing to be wrong on a host
billed by the second.
Untested here. FP8 scores 10/10 on the ten-task probe against 4-bit's 9/10 — but that same probe scored BF16 and 4-bit identically while their greedy prose diverged at word 33 and shared 24.4% of its words. That is noise, not evidence.
Do not cite this work as evidence that FP8 is more faithful
than NVFP4. The instrument that would settle it — KL divergence and top-1 agreement
against a BF16 reference, three arms on one pod — ships in the repo as
eval/kl-eval.py and has not been run. It would take about 40 minutes and $5.
Everything above came off the committed scripts. Nothing was measured by hand.
cp .env.example .env # RunPod key, HF token, EPHEMERAL Tailscale key
./check-requirements.sh # runs entirely before anything bills
./up.sh # rent, join the tailnet, serve
./benchmark.sh # the sweep — copied to the pod and run there
./down.sh # and prove nothing is still billingRaw JSON for every figure on this page is in results/, and each file carries the
engine's own startup lines so the configuration behind a number stays checkable after the pod is
gone. The reasoning lives in the docs/ bundle.