# Serving-engine concurrency bench: GGUF llama.cpp vs vLLM NVFP4 under fan-out (2026-08-20)

> **CORRECTION (2026-08-20, same day, added after re-bench). The verdict below is
> RETRACTED. The vLLM side of the original bench was misconfigured** and its numbers
> are not representative. The original run launched vLLM with a hand-rolled command
> that OMITTED `--kv-cache-dtype turboquant_4bit_nc` and `--speculative-config mtp`
> (NVFP4 weights but no MTP speculative decoding and no TurboQuant KV), while
> the correct, tuned recipe was sitting patched-and-ready in our own
> `tools/qwen38-nvfp4-vllm` (== github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090).
>
> Re-benched with the full recipe (MTP-3 + TurboQuant 4-bit KV + the PR #40914
> garble patch). Corrected numbers (`serving-conc-vllm-MTP3-recipe.txt`):
>
> | N | GGUF Q5+MTP | vLLM (orig, crippled) | **vLLM full recipe** |
> |---|---|---|---|
> | 1 | 136 | 57 | **145** |
> | 2 | 185 | 108 | **275** |
> | 4 | 320 | 209 | **613** |
> | 8 | 429 | 436 | **1080** |
>
> **vLLM NVFP4 with the proper recipe wins at EVERY concurrency level, not just
> high-N**: 145 vs 136 single-stream (and snappier: 3.8-4.5s vs 5.6s p50), and
> **2.5x GGUF at N=8 (1080 vs 429 agg tok/s)**. Per-request stays flat ~135 tok/s
> through 8-way (continuous batching + MTP both working); output stayed coherent
> under load ("the quick brown fox" clean, #40914 patch holds); the unit did not
> crash. The recipe's own "MTP+concurrency crashes this KV path" note did NOT
> reproduce on sm_120 at max-num-seqs 8 / 32K ctx. N=8 is the sweet spot (=
> max-num-seqs); N=12/16 merely oversubscribe the 8 slots.
>
> **Corrected practical verdict:** on raw throughput at all tested concurrency,
> vLLM-NVFP4 (MiaAI recipe) beats GGUF llama.cpp. GGUF/llama-swap remains the
> daily-driver default for its OTHER advantages: Q5 quality (NVFP4 is -8.9 pts
> polyglot pass@2), vision, desktop-up flexibility, multi-model swap, no ~90s cold
> start. Stand up vLLM-NVFP4 when the box is headless AND sustained fan-out
> throughput is the goal. Everything below the line is the original (wrong) reading,
> kept for the record.
>
> ---

> **UPDATE 3 (2026-08-21): DFlash2 replaces DSpark in the fast lane. The N<=4
> column has a new owner.** inco.ai released DFlash 2 (parallel block-drafting
> spec decode; Karakeep save -> `https://inco.ai/blog/dflash2/`) with an official
> 2B BF16 drafter for Qwen3.8-27B (`incoai/Qwen3.8-27B-DFlash2`, pure-safetensors,
> safety-reviewed). Support landed on SGLang main AFTER v0.5.18's cut, so the lane
> now runs `lmsysorg/sglang:dev` @ f825d72 (2026-08-20), retagged locally as the
> immutable `lmsysorg/sglang:dflash2-f825d72`. Launch pins from sglang PR #35825's
> verified RTX 5090 grid: `--mem-fraction-static 0.91 --chunked-prefill-size 1024`,
> bf16 SSM state (float32 SSM does not fit beside a draft model on this card).
> Raw: `serving-conc-sglang-dflash2.txt`. Same target checkpoint, same harness:
>
> | N | SGLang+DSpark (old lane) | **SGLang+DFlash2 (new lane)** | vLLM+MTP3 |
> |---|---|---|---|
> | 1 | 227 | **295** | 145 |
> | 2 | 505 | **617** (308/req) | 275 |
> | 4 | 520 (bs=2 cap) | **657** | 613 |
> | 8 | n/a | 654 | **1080** |
>
> DFlash2 is +30% single-stream over DSpark AND removes DSpark's hard bs=2 cap:
> it now beats vLLM+MTP3 at N=4 too, plateauing ~655 aggregate. vLLM+MTP3 keeps
> only the N>=8 sustained-saturation crown. PR #35825's TPOT numbers at 8k-deep
> context (DFlash2 4.92 ms vs DSpark 7.41 ms, accept 4.29 vs 2.59) corroborate
> the ratio holding at depth. Smoke-tested through the lane: coherent prose,
> correct structured tool_calls, exact math (spec decode is verify-exact, so
> output quality equals the target model by construction). Validated end-to-end
> via llama-swap (`qwen38-fast`): 292/614, cold start ~80s, clean teardown to
> 524 MiB. Config + wrapper updated 2026-08-21; DSpark recipe retired.
>
> Open follow-up: llama.cpp PR #27342 adds DFlash2 GGUF drafting (1.1 GB Q4_K_M
> drafter, accept 5.39) - if merged (or side-built), the swappable GGUF lane
> itself could reach fast-lane speeds while keeping vision + hot-swap.
>
> ---

> **UPDATE 2 (2026-08-20 evening): SGLang added, serving-engine axis now CLOSED.**
> Benched SGLang (`lmsysorg/sglang:qwen38-27b`) on the same box with the tuned
> ModelOpt NVFP4 checkpoint (`gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090`) and
> its matching **DSpark** drafter. Raw: `serving-conc-sglang.txt`. NInfer was
> deliberately dropped (from-scratch C++/CUDA build, highest setup risk, mid-pack
> per the community spectrum). Full four-way aggregate tok/s picture:
>
> | N | GGUF Q5+MTP | vLLM+MTP3 | SGLang no-spec | **SGLang+DSpark** |
> |---|---|---|---|---|
> | 1 | 136 | 145 | 85 | **227** |
> | 2 | 185 | 275 | 167 | **505** |
> | 4 | 320 | **613** | 327 | 520 (capped) |
> | 8 | 429 | **1080** | 638 | n/a (cap 2) |
>
> **The verdict splits cleanly by concurrency, and there is no single winner:**
> - **N=1-2 (interactive / light fan-out): SGLang+DSpark dominates** (227 single,
>   505@2). DSpark is speculation like MTP but stronger here, and at N=2 it compounds
>   with batching (252 tok/s PER request). Nothing else is close. Latency is also best
>   (2.8s p50). But the qwen38-27b build **caps DSpark concurrency at bs=2** (it clamps
>   `--max-running-requests` to 2), so it is a low-concurrency tool by construction.
> - **N=4+ (sustained heavy fan-out): vLLM+MTP3 wins** (613@4, 1080@8). Past DSpark's
>   2-slot ceiling, vLLM's MTP-over-continuous-batching pulls ahead and keeps scaling.
> - **SGLang no-spec** scales cleanly (flat 84 tok/s per-req) but never wins a column:
>   without speculation it trails vLLM+MTP3 at every N.
> - **GGUF/llama-swap** is mid-pack on raw speed but keeps what the NVFP4 engines give up:
>   Q5 quality (NVFP4 -8.9 pts polyglot), vision, desktop-up, multi-model hot-swap.
>
> **Crossover: DSpark wins N<=2, vLLM+MTP3 wins N>=4.** For Hermes fan-out the practical
> rule: a FEW heavy concurrent agents (1-2) -> SGLang+DSpark (blazing, 227/505); MANY
> sustained agents (4-8+) -> vLLM+MTP3 (1080@8). Both are headless-class (~29 GiB), so
> GGUF/llama-swap remains the desktop-up daily default. Neither NVFP4 engine is a
> drop-in for the swap ecosystem; stand one up deliberately for a throughput workload.
>
> ---

Board card `task_20260819_a-6YOc`. The question the single-stream NVFP4 trial
could not answer: does vLLM's continuous batching beat llama.cpp under
*concurrent* load (Hermes agent fan-out), enough to overcome NVFP4's known
single-stream speed + quality deficit?

Harness: `benchmarks/concbench.py` (N identical chat requests fired at once,
threads; measures aggregate tok/s = total completion tokens / batch wall-clock,
per-request tok/s, and p50/p95 latency). Same model family (Qwen3.8-27B), same
prompt, N = 1/2/4/8. **Both engines capped at 8-way** (GGUF `--parallel 8`,
vLLM `--max-num-seqs 8`), so N=8 is the max meaningful point here.

- **GGUF**: llama.cpp, Q5_K_M + embedded MTP, `--parallel 8`, `-c 65536`
  (8k/slot), continuous batching (default on), via llama-swap. 27.7 GiB.
- **vLLM**: RadixArk NVFP4 weights, `--max-num-seqs 8`, `--max-model-len 16384`,
  no MTP / no TurboQuant-KV (the trial's single-session recipe hardcodes
  `--max-num-seqs 1` because "MTP + concurrency crashes this KV path" - the
  concurrency profile is legitimately different). 25.5 GiB, 84s cold start.

## Results

| N | GGUF agg tok/s | vLLM agg tok/s | GGUF per-req | vLLM per-req | GGUF p50 lat | vLLM p50 lat |
|---|---|---|---|---|---|---|
| 1 | **136.0** | 56.9 | 136 | 57 | **5.6s** | 13.9s |
| 2 | **184.8** | 107.6 | 93 | 54 | 7.6s | 13.2s |
| 4 | **319.8** | 208.5 | 89 | 52 | 8.5s | 13.7s |
| 8 | 429.2 | **436.2** | 61 | 55 | 10.9s | 12.6s |

Raw: `serving-conc-gguf.txt`, `serving-conc-vllm.txt`.

## Reading

1. **GGUF wins single-stream by 2.4x (136 vs 57) and stays ahead through N=4.**
   The crossover is right at **N=8**, where they tie (429 vs 436, within noise).
2. **The scaling *shapes* are the real finding, and they favor vLLM's
   architecture past the crossover.** vLLM's per-request throughput is FLAT
   (~52-57 tok/s at every N) - the textbook continuous-batching signature: it
   serves 8 concurrent at the same per-request speed as 1, so aggregate scales
   ~linearly (57 -> 436 = 7.7x over 8x load). GGUF's per-request COLLAPSES
   (136 -> 61) as its cruder batching shares compute less efficiently, so its
   aggregate saturates (already flattening: 320 -> 429 from N=4->8). Extrapolated,
   vLLM pulls clearly ahead beyond N=8; GGUF is near its ceiling.
3. **Latency is the other axis, and it favors GGUF at low concurrency.** GGUF is
   5.6s at N=1; vLLM sits at a ~13s FLOOR regardless of N (everything rides the
   batched pipeline). For interactive / low-concurrency use, GGUF is far snappier.
4. **MTP fades under load, as hypothesized.** GGUF single-stream 136 (MTP helping)
   collapses to 61/req at N=8 - the draft head's "fill idle compute" win vanishes
   once the batch saturates the GPU. This is why dropping MTP for vLLM's batched
   path costs nothing at high N.

## Verdict for Hermes fan-out

**It depends on the concurrency profile, and the honest answer splits at N=8:**

- **Bursty / low-to-moderate concurrency (agents not all generating at once, the
  common case): GGUF `--parallel` wins.** 2.4x faster single-stream, much better
  latency, and it keeps everything vLLM gives up: Q5 quality (NVFP4 is -8.9 pts
  polyglot pass@2, `REPORT-2026-08-16-nvfp4-lane-trial.md`), vision, the MTP
  speedup when not saturated, desktop-up capability of the wider entry set, and
  co-residence in the llama-swap ecosystem. No 84s cold start, no headless-only.
- **Sustained high concurrency (N >= 8, many agents generating continuously):
  vLLM/NVFP4 begins to win on aggregate throughput** and would widen past N=8, at
  the cost of quality, a ~13s latency floor, headless-only (25.5 GiB), and no
  vision/MTP. Worth it only if Hermes fanning genuinely sustains 8+ concurrent
  generations.

**Recommendation: do NOT stand up a permanent NVFP4 vLLM entry yet.** For
realistic bursty fan-out, GGUF `--parallel` is the better tool and needs only a
config flag (bump the daily/desktop entries' `--parallel`, headless). Revisit
vLLM if Hermes fanning is measured to sustain 8+ concurrent agents - that's the
open design question the card flagged.

## Caveats / follow-ups

- **N>8 uncharacterized**: both engines capped at 8-way. Relaunching both at 16
  (GGUF `--parallel 16`, vLLM `--max-num-seqs 16`) would nail the post-crossover
  slope - the single most useful follow-up if fan-out will run heavy.
- **Throughput-only, not quality.** The reasoning-vs-answer token mix differed
  (vLLM here ran without a reasoning parser), so tok/s is the fair metric; the
  quality delta is the trial's measured -8.9 pts, not re-measured here.
- **vLLM is headless-class** (weights alone 20.2 GiB). This bench relied on the
  desktop being dormant (524 MiB); a screen wake mid-run could OOM. Confirms the
  earlier read: NVFP4 vLLM is a headless-throughput tool, not a desktop-up one.