HermiHg/Qwen3.8-Flash-Next-Optimized-MTP-Heads-exl3

Qwen3.8-Flash-Next Optimized MTP Head (EXL3)

A per-group mixed-precision EXL3 quantization of the Qwen3.8-Flash-Next MTP (multi-token prediction) draft head, packaged as a sidecar that shadows the built-in MTP weights of the base model. Flash-Next is a mixture-of-experts model, so its single MTP layer is dominated in size by the routed-expert FFN (512 experts, ~97% of the head's quantized parameters). Each optimized checkpoint is far smaller than the same-acceptance uniform quant — because that expert FFN tolerates very low bit widths (its floor is ~2 bpw) and gets the bulk of its size budget cut there, while the small, quality-critical attention and input-projection groups (≤ 3% combined) are held high.

This is not a standalone model. It is the draft head a speculative-decoding server runs alongside the target: the MTP head proposes tokens, the target verifies them, and decoding stays lossless (accepted output is identical to the target's own output). What this changes is speed and memory — a smaller, higher-acceptance draft head means more tokens accepted per verification step and less VRAM for the drafter.

What the sidecar does

The files in this repo contain only the mtp.* tensors — the MTP head: the two input projections (fc_embedding, fc_hidden), one full-attention block (q/k/v/o plus the sparse-attention indexer), the always-on shared expert (gate/up/down), and the 512 routed experts (fused gate+up and down). Norms, hyper-connections and the router gates are kept in bf16. When placed alongside a base model, exllamav3 loads the directory's safetensors in alphabetical order, and tensors in a later file shadow earlier ones. A file named mtp-*.safetensors sorts after the base model's model-*.safetensors, so this sidecar's mtp.* tensors override the base model's built-in MTP head.

Tip — trim the base model: because the sidecar shadows the base model's own MTP weights, those original mtp.* tensors are dead weight. You can ask your agent to strip the MTP weights from any base model you pair this sidecar with, reclaiming their full size with no change in behavior.

Compatible base models

The MTP head is defined by the shared Qwen3.8-Flash-Next base, so this sidecar works with any model built on that base (Qwen3.8-Flash-Next and its derivatives).

Checkpoints

All four release checkpoints hold the small groups high (input_fc=8, indexer=8, shared=8) and vary only the attention and routed-expert FFN — where the size budget actually lives. gu = fused routed gate+up, d = routed down.

File	Size	eff. bpw	config (attn / gu / d)
mtp-1.8bpw.safetensors	549 MiB	1.8	5 / 1.5 / 1.5 — smallest
mtp-2.3bpw.safetensors	705 MiB	2.3	6 / 2.0 / 2.0 — small
mtp-2.6bpw.safetensors	805 MiB	2.6	6 / 2.0 / 3.0 — recommended
mtp-3.1bpw.safetensors	955 MiB	3.1	6 / 2.5 / 3.5 — largest

Two uniform quantizations — K4 (all groups @ 4, ≈ 1.3 GiB, 4.2 bpw) and K8 (all groups @ 8, ≈ 2.5 GiB, 8.1 bpw) — are used as the size–quality baselines in the plots and comparison below.

Measured performance

Draft acceptance (accepted draft tokens / total draft tokens), 25 runs per workload, temperature=0, n=4 draft tokens per step, against the Qwen3.8-Flash-Next target. Three workloads: repetitive, prose, code_edit. geomean is the geometric mean across the three — a size-balanced overall figure that weights every workload equally.

Checkpoint	Size	repetitive	prose	code_edit	geomean
mtp-1.8bpw	549 MiB	0.868	0.329	0.977	0.654
mtp-2.3bpw	705 MiB	0.888	0.339	0.980	0.666
mtp-2.6bpw	805 MiB	0.891	0.352	0.977	0.674
mtp-3.1bpw	955 MiB	0.880	0.350	0.978	0.670
uniform K4	≈ 1.3 GiB	0.898	0.337	0.979	0.666
uniform K8	≈ 2.5 GiB	0.897	0.347	0.981	0.673

Mean accept length (draft tokens accepted per verification round, n=4) — the direct throughput driver (higher = fewer verification steps per token):

Checkpoint	Size	repetitive	prose	code_edit	geomean
mtp-1.8bpw	549 MiB	3.47	1.32	3.91	2.61
mtp-2.3bpw	705 MiB	3.55	1.36	3.92	2.66
mtp-2.6bpw	805 MiB	3.56	1.41	3.91	2.70
mtp-3.1bpw	955 MiB	3.52	1.40	3.91	2.68
uniform K4	≈ 1.3 GiB	3.59	1.35	3.92	2.67
uniform K8	≈ 2.5 GiB	3.59	1.39	3.92	2.69

The mixed-precision win

The optimized checkpoints sit on the size–quality Pareto frontier and dominate a uniform quant that is the same size or larger:

• mtp-2.6bpw (805 MiB) beats uniform K4 (4.2 bpw, ≈ 1.3 GiB) on geomean (67.4 vs 66.6) while being 39% smaller.
• mtp-2.6bpw (805 MiB) matches uniform K8 (8.1 bpw, ≈ 2.5 GiB) on geomean (67.4 vs 67.3) while being ~⅓ the size.
• mtp-3.1bpw adds no headroom over 2.6bpw (geomean 67.0 vs 67.4), and mtp-1.8bpw costs ~2 pp geomean to save 256 MiB.

No uniform quant at any size is both smaller and better: K4 is larger and weaker; K8 is 3× larger for no geomean gain.

Acceptance vs size — Pareto frontier (n=4 draft)

Each plot shows draft acceptance against effective bits/weight. The black line connects only the Pareto-optimal checkpoints; a checkpoint is dominated (gray open) if a smaller one is at least as good. Green = optimized release, blue = uniform reference.

Acceptance vs size (n=7 draft) — release candidates

A second set at n=7 draft tokens per step (longer prefix) for the release candidates. Absolute acceptance is lower than the n=4 set because a longer prefix includes harder late-draft tokens — the two sets are not directly comparable — but the ranking holds: 2.6bpw is the geomean leader, 3.1bpw second, 2.3bpw third.

How it was built

Quantized clean from the BF16 MTP head (no dequant-from-quant) with exllamav3's EXL3 format: per-group bitrate, identity Hessian (no calibration data), seed 0, mul1. Norms, hyper-connections and router gates are held in bf16 (a bit-exact copy of the source). The six quantizable groups are allocated independently:

Group	Params	Share	1.8bpw	2.3bpw	2.6bpw	3.1bpw	Sensitivity
input_fc (fc_embedding + fc_hidden)	13.1 M	0.5%	8	8	8	8	highest — held at 8; 16-bit gives no gain
attn (q/k/v/o)	49.8 M	1.9%	5	6	6	6	high — floor ~4, geomean peaks at 6
indexer (index_qk_proj)	1.6 M	0.06%	8	8	8	8	low — insensitive, 4–16 flat
shared (shared expert gate/up/down)	4.9 M	0.19%	8	8	8	8	medium — flat 5–8, cliff below 5
egateup (routed experts, fused gate+up)	1,678 M	64.9%	1.5	2.0	2.0	2.5	floor ~2.0; K=1.5 is a soft edge
edown (routed experts, down)	838.9 M	32.4%	1.5	2.0	3.0	3.5	floor ~2.0; K=3 a small marginal gain
Total (EXL3 linears)	2,586 M	100%	1.8	2.3	2.6	3.1

The allocation comes from a systematic per-group sensitivity sweep: each group's bit width was lowered one at a time (the rest held at baseline) and the acceptance cost measured. The protection priority that emerged is

input_fc  >  attn  ≈  shared  >  egateup  ≈  edown  >  indexer

input_fc is the most sensitive group and is the last to be cut, but 16-bit buys no measurable gain over 8-bit, so 8 is its ceiling. attn is second — it has a floor near 4 and its geomean peaks at 6. shared is flat from 5 to 8 with a cliff below 5. indexer is too small to bother compressing, but K=8 is free. The routed-expert FFN (egateup + edown, ~97% of the head) has a floor near 2.0 bpw, which is what makes the whole head compressible to ~2–3 bpw without loss: the small sensitive groups are held high (cheap, ≤ 3% of the head) and the huge expert FFN carries the size savings. That reallocation is exactly what lets 2.6 bpw match the 8.1 bpw uniform head at a third of the size.

Usage

Pair the sidecar with any compatible base model (see above) and run exllamav3 with MTP drafting enabled:

1. Copy one of the mtp-*.safetensors files into the base model's directory. Keep the mtp- prefix so it sorts after the base model's model-*.safetensors and shadows the built-in head.
2. (Optional) strip the base model's own mtp.* tensors to reclaim their drive space.
3. Launch with --mtp or configure mtp drafting in tabbyAPI.

License

Apache-2.0. Base model: Qwen/Qwen3.8-Flash-Next.