MiMo-V2.6-Distill-Qwen-9B-MTP-GGUF

MiMo-V2.6-Distill-Qwen-9B GGUFs with a trained multi-token-prediction (MTP) draft head merged in as block 32, for self-speculative decoding with --spec-type draft-mtp in the llama.cpp Qwen3.5 fork (prism).

The base model ships an MTP config field (mtp_num_hidden_layers: 1) but zero MTP weights — this repo trains them, so the model can draft against itself with no second model.

Files

File	Size	Notes
MiMo-V2.6-Distill-Qwen-9B-Q5_K_M-MTP.gguf	6.3 GiB	Recommended. Q5_K_M cut from BF16 with a custom imatrix; loop-safe chat template embedded
MiMo-V2.6-Distill-Qwen-9B-Q4_K_M-MTP.gguf	5.7 GiB	bartowski Q4_K_M base + MTP head (Q8_0)
MiMo-V2.6-Distill-Qwen-9B.imatrix.gguf	5.2 MB	The importance matrix — cut other tiers from BF16 with llama-quantize --imatrix
chat_template.jinja	4 KB	The MiMo thinking template (pre-fills the reasoning opener). Embedded in the Q5 file; pass --chat-template-file when serving the Q4 file

The MTP head: 243.3M params, Q8_0 linear weights + F32 norms, initialized from Qwen/Qwen3.5-9B shipped mtp.* tensors and fine-tuned on MiMo (~32.7M tokens, loss 1.7 → 0.5). Base GGUF tensors are preserved byte-for-byte.

Usage (llama.cpp Qwen3.5/prism fork)

# Q5 (template embedded — nothing extra needed)
llama-server -m MiMo-V2.6-Distill-Qwen-9B-Q5_K_M-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngl 99 -fa on -c 8192 --jinja

# Q4 (pass the template explicitly)
llama-server -m MiMo-V2.6-Distill-Qwen-9B-Q4_K_M-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngl 99 -fa on -c 8192 --jinja \
  --chat-template-file chat_template.jinja

--spec-type draft-mtp requires a build with the Qwen3.5 MTP graph (graph_mtp) — current prism/prism-ML forks have it. Stock builds load the file fine without it; generation still works.

Measured results (RTX 4090, -fa on -ngl 99)

Config	Acceptance	Mean draft	Spec on	Spec off	Speedup
Q4_K_M, repetitive greedy (256 tok)	85.1 % (160/188)	2.70	215.7 t/s	135.6 t/s	1.59×
Q5_K_M+imatrix, repetitive greedy (256 tok)	86.1 % (161/187)	2.71	214.2 t/s	—	~1.6×
natural text, temp 0.4 (96 tok)	38.0 % (41/108)	1.76	148.9 t/s	132.3 t/s	1.13×

Greedy/near-greedy decoding benefits most (the draft targets the argmax); sampled text accepts less, as with any speculative method — the gain shrinks toward break-even as temperature rises.

Quality (perplexity, 400×512-token code-heavy slice of the calibration corpus, lower is better):

Q5_K_M + imatrix + MTP:  5.1550 ± 0.043
Q4_K_M (bartowski):      5.2770 ± 0.045

Offline parity check: the head's top-1 proposal probability matches a pure-PyTorch replication of the runtime graph to 3 decimals (e.g. jumps p=0.987 live vs 0.988 offline).

The imatrix (Q5 tier)

Recipe follows Tiel-Inspired-Coder-9B-GGUF: importance matrix measured on a Q8_0 of the BF16 weights, no weight changes. Calibration corpus from eaddario/imatrix-calibration (MIT), interleaved 3:1 code_medium : combined_all_large, 3,000 chunks × 512 tokens (~1.5M tokens). Q8_0 reference PPL on the full corpus: 7.0682 ± 0.022. The matrix file ships in this repo.

The chat template matters

The template embedded in the Q5 file (also chat_template.jinja, adapted from coder543's MiMo gist) pre-fills the thinking opener to establish the reasoning boundary. The stock template lets this model's reasoning mode degenerate into deterministic repetition loops — greedy thinking runs burned 14k+ tokens without ever emitting an answer. With this template, greedy + thinking finishes cleanly (744 reasoning chars, finish: stop). If a build loops in thinking mode, check the template first.

Notes for people re-merging their own head

• HF Qwen3_5RMSNorm applies gain (1 + w); ggml multiplies plain w. The official converter bakes the +1 (conversion/qwen.py: name.endswith("norm.weight") -> data + 1, MTP names included). Write w + 1.0 for every 1-D norm at merge time or draft acceptance is exactly 0.00000 while everything else still "looks right".
• 1-D norms must be F32 to match the base file's dtype (F16 → mul: dst f32, src0 f32, src1 f16 crash).
• The MTP layer needs recurrent_layers extended with False and nextn_predict_layers = 1 (arch-prefixed, not general.).
• Converting from HF when the config declares mtp_num_hidden_layers: 1 but ships no mtp weights yields block_count=33 + nextn=1 without blk.32 tensors and no recurrent_layers — normalize to block_count=32 + a 32-entry GDN pattern before merging.

Credits

• Base model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT)
• Base GGUF: bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF
• MTP init weights: Qwen/Qwen3.5-9B
• imatrix + template methodology: VitreousCut/Tiel-Inspired-Coder-9B-GGUF
• Runtime: llama.cpp (prism fork) with graph_mtp / --spec-type draft-mtp

MIT (same as the base model).