# Qwen3.8-27B UD-IQ3_XXS quant-ladder verification (2026-08-17)

Board card `task_20260816_B4ORYK`: a Karakeep-saved community claim (witcheer on
X) said Qwen3.8-27B's quant tax is near zero, with UD-IQ3_XXS holding 92.7 of
Q8_0's 93.7 (their eval) from an 11.1 GiB file. Community numbers had twice
failed to reproduce on this box (draft tuning, n-max 2), so the rule was:
measure, don't import.

**VERDICT: the claim survives our whole suite. IQ3 promoted to the desktop
entry, which now carries the FULL headless feature set (131k ctx + MTP +
vision) desktop-up, at less VRAM and more speed than the old 32k UD-Q4 entry.**

## Method

Test entry `qwen3.8-27b-iq3test` (temporary, since promoted) at the HEADLESS
config shape: 131072 ctx, q8_0 KV, embedded-MTP `--spec-type draft-mtp`,
vision mmproj, model-card sampling (temp 1.0 / top-p 0.95 / top-k 20). Weights:
`unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-IQ3_XXS.gguf` (11.9 GB file).

Four gates: 25-task bakeoff (`run_qwen38_bakeoff.sh iq3`), structured
tool-calling (`toolcall_probe.py`, new, scripts the check the 08-14 deployment
did ad hoc), vision red-square probe, and `longctx_probe.py` at 2k/60k/110k
fills with seed 1 (seed-matched to the Q5_K_M baseline runs).

## Results

| gate | UD-IQ3_XXS | reference |
|---|---|---|
| bakeoff | 25/25, median 117 tok/s, 35s, 4,083 tok | UD-Q4 desktop 25/25 / 110 / 47s / 3,979; Q5 headless 25/25 / 97 / 48s / 3,862 |
| tool calls | 5/5 finish_reason=tool_calls, clean JSON args | same check passed at Q5/Q4 |
| vision | red-square pass | same |
| longctx 110k (seed 1) | **1.000** | Q5_K_M: 0.976 |
| longctx 60k (seed 1) | 0.833 (1 agg DNF) | Q5_K_M same run: **0.833 (1 agg DNF)** - identical |
| temp-0 code probe | 162 tok/s, 62.5% draft acceptance | UD-Q4 desktop ~146 tok/s |
| VRAM resident | **~20.8 GiB at 131k ctx** | old desktop entry 21.8 GiB at 32k |

Raw: `bakeoff-qwen38-iq3.jsonl`, `longctx-qwen38-iq3-s1.jsonl`,
`longctx-qwen38-iq3-s1-60k-retry.jsonl`, `longctx-qwen38-q5-s1-60k.jsonl`.

## Findings

1. **No detectable quant tax on any gate.** Token spend sits inside the
   Q4/Q5 family band (4,083 vs 3,862-4,083), so no degradation-driven
   verbosity either. IQ3 is the FASTEST qwen3.8 leg yet run (smaller weights
   = faster MTP verify passes).
2. **The 60k longctx dip is a probe artifact, attributed properly this time.**
   Both the 8192- and 16384-budget IQ3 runs DNF'd the seed-1 60k aggregation
   (model thinking past budget at temp 1.0), while the SAME probe at 110k
   scored 1.000 with aggs completing in 790-6,502 thinking tokens. The
   clincher: Q5_K_M on the identical seed-1 60k fill produced the identical
   single-agg-DNF 0.833. Overthinking on that task is a model+seed property,
   not a quant effect.
3. **The win is bigger than the card projected.** The card hoped to shed ~5 GB
   from the 32k desktop entry. Measured: 11.1 GiB weights buy 131k ctx + MTP
   + vision in ~20.8 GiB, i.e. the desktop compromise (no long ctx) is gone
   entirely, with ~2.8 GiB margin under the ~9 GB GNOME tax.
4. **Honest limit: the suite is saturated.** Every frontier-27B config sweeps
   it, so a 1-point MMLU-Pro-class tax (the claim's own 92.7 vs 93.7) is
   invisible here. That is why IQ3 takes the desktop-FALLBACK slot while the
   daily drivers (`qwen-best` text, `agent-best` agentic) stay on Q5_K_M
   headless. Before ever making IQ3 front daily work, run a harder
   discriminator (aider polyglot slice, recipe in HANDOFF-agent-shootout.md
   3b).

## Round 2: hard-discriminator vetting (same day, Matt's call)

The saturated-suite caveat got its answer the same day: aider polyglot (the
shootout harness, identical settings, quant as the only variable) plus the
SkiFree one-shot, with a 2x2 variance design after run 1 looked ambiguous.

### Aider polyglot, 34 Python exercises, whole format, effort medium

| run | pass@1 | pass@2 | completion tok | s/case |
|---|---|---|---|---|
| Q5 run A (08-16 baseline) | 10/34 | 26/34 | 173.6k | 43.7 |
| Q5 run B (control rerun) | 10/34 | 29/34 | 160.2k | 40.7 |
| IQ3 run A | 11/34 | 23/34 | 225.0k | 45.5 |
| IQ3 run B | 8/34 | 26/34 | 226.6k | 46.0 |

- **Correctness: overlapping, mild negative lean.** Q5 drew {26, 29}, IQ3
  {23, 26} - IQ3's best equals Q5's worst; a ~3-exercise mean gap at n=2
  with +-3 within-side spread is not separable from sampling noise. The
  per-exercise diff (run A vs baseline) was churn in both directions,
  concentrated in attempt-2 repair coin-flips (IQ3 won bowling/dominoes
  outright, lost bottle-song/hangman/zipper).
- **Token economy: the robust finding.** IQ3 spent +35% completion tokens
  in BOTH runs (225-227k vs 160-174k, non-overlapping, tight per side),
  erasing its decode-speed edge (46 vs 41-44 s/case). More thinking and
  more repair for slightly-worse-leaning results = the shape of a mild
  real quant tax on hard tasks. 100% well-formed on every run - no
  formatting/instruction degradation.

### SkiFree one-shot (human judge: Matt; headless CDP play-test: agent)

Both quants produced complete, playable, zero-console-error games with the
full loop verified end-to-end (steer/tuck, obstacles, wipeouts, yeti chase
past threshold, catch, game-over screen, localStorage high score, restart) -
screenshots in `skifree/playtest/`. Q5's build ("The Eternal Slope") is the
richer artifact: web-audio sound effects, yeti proximity meter, attract-mode
demo, crash i-frames. IQ3's ("Abominable Snow Chase") is mechanically
complete with a plainer look plus its own touches (blur auto-pause, in-game
yeti threshold hint). Matt's verdict: Q5 looks better; IQ3 still very
impressive. Single sample each at temp 1.0 - aesthetics are noisy; the
directional agreement with polyglot is what counts.

### Round-2 verdict

The original slotting SURVIVES vetting, now with evidence instead of a
caveat: **IQ3 stays the desktop-fallback entry** (its job is full features
desktop-up, where it is strictly better than the old UD-Q4/32k entry), and
**the Q5 daily drivers stay Q5**. On frontier-difficulty work IQ3 pays a
real, small efficiency tax (+35% thinking tokens) and leans a few points
lower on repair-round solve rate. "Near-lossless" holds for
capability-in-kind; token-economy-lossless it is not.

Raw: `tools/aider-bench/tmp.benchmarks/2026-08-17-*` (q38-iq3-py,
q38-q5-rerun-py, q38-iq3-run2-py), `skifree/stats.jsonl`,
`skifree/playtest/*-playtest.json`.

## Round 3: IQ4_XS buys the token tax back (2026-08-19)

Round 2 left IQ3 as the desktop entry with one known weakness: the +35%
completion-token tax on hard tasks. Round 3 asked whether a higher-quality
quant that still fits desktop-up removes it. It does.

Candidate: unsloth `Qwen3.8-27B-IQ4_XS.gguf` (15.7 GB) - unsloth lineage
chosen so the embedded MTP head (`blk.64.nextn.*`) is present, same as the
IQ3/Q4/Q5 entries. Vetted at the desktop config shape.

**Fit (the trade is context, not features):**

| config | VRAM resident | desktop-up (~23.6 GiB budget) |
|---|---|---|
| IQ3_XXS @ 131k (round 2) | 20.8 GiB | fits, 2.8 GiB margin |
| IQ4_XS @ 131k | 24.7 GiB | OVER budget |
| IQ4_XS @ 64k | 21.9 GiB | fits, 1.7 GiB margin |
| IQ4_XS @ 48k | ~20.5 GiB | fits, ~3 GiB margin (fallback) |

**Quality gates:** 25/25 bakeoff (117 tok/s), 5/5 tool calls, vision
red-square pass, MTP 70% draft acceptance (164 tok/s temp-0). All intact.

**The discriminator - 2x2 aider polyglot, same harness:**

| quant | pass@2 | completion tokens |
|---|---|---|
| Q5_K_M (daily) | 26, 29 | 160k, 174k |
| IQ3_XXS (round 2) | 23, 26 | 225k, 227k |
| **IQ4_XS** | **26, 28** | **166k, 162k** |

IQ4's token spend is tight, non-overlapping with IQ3, and squarely in Q5's
range - the +35% tax is gone. pass@2 {26,28} is >= IQ3 and ties Q5. So IQ4
is strictly better than IQ3 on the metric that distinguished them (token
economy), marginally better on solve rate, at the cost of 131k -> 64k
context. For the desktop-up FALLBACK role (interactive work while the PC is
up; >64k goes headless) that trade clearly favors IQ4: the token economy is
felt on every hard task, the context loss only on rare >64k desktop-up
sessions.

**Verdict: IQ4_XS/64k promoted to the desktop entry** (was IQ3_XXS/131k).
The IQ3 promotion (rounds 1-2) was correct given what was tested then;
round 3 supersedes it. Raw:
`tools/aider-bench/tmp.benchmarks/2026-08-19-*` (q38-iq4-py, q38-iq4-run2-py),
`benchmarks/results/bakeoff-qwen38-iq4.jsonl`.

## Config changes shipped

- `llama-swap/config.yaml`: `qwen3.8-27b-desktop` swapped UD-Q4_K_XL/32k ->
  UD-IQ3_XXS/131k (alias `qwen38-desktop` unchanged; clients unaffected).
  Temporary `qwen3.8-27b-iq3test` entry removed. Verified serving end-to-end
  through llama-swap after the swap (health, VRAM, temp-0 timings identical
  to the test entry).
- `benchmarks/run_qwen38_bakeoff.sh`: explicit `iq3` leg kept for re-runs.
- `benchmarks/toolcall_probe.py`: new reusable tool-call gate.
- `Qwen3.8-27B-UD-Q4_K_XL.gguf` kept on disk as the revert path.