scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp

This is mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit at 8284ab1029a2b0114824175aa7f04740ff318934 with the multi-token-prediction (MTP) head of Qwen/Qwen3.5-122B-A10B put back. The OptiQ conversion drops that head, and speculative decoding needs it.

The head is inline, as in the official checkpoint and mlx-community/Qwen3.5-122B-A10B-oQ4-mtp. Its 46 tensors are named language_model.mtp.*, live in model-mtp.safetensors (1.32 GiB), and are listed in model.safetensors.index.json. Every quantized head module has its own quantization/quantization_config entry. Total on disk is 44.1 GiB, against 42.8 GiB without the head.

Everything else is byte-identical to the OptiQ repo. That covers the nine weight shards, the vision sidecar, the tokenizer, and every existing config, index and quantization entry. The copy was sha256-checked file by file. mlx-lm's qwen3_5_moe sanitize drops language_model.mtp.*, so the model loads and runs exactly as the OptiQ original does.

Speculative decoding with its head

M5 Max, scratchy (PR #419), scr chat --bench, median of three rounds alternated with plain decode:

tok/s TTFT ITL p50
plain decode 97.3 91.2 ms 10.29 ms
drafting with this head (2 drafts a step) 143.9 (+48%) 94.8 ms 18.67 ms

The head drafts 2.72 tokens a verify step, 86.2% of drafts accepted (162 of 188; scr serve, one greedy request, 256 tokens).

Greedy output with drafting matches plain decode until a single near-tie token, where it takes the other choice; each path is exactly reproducible. The verify step runs the target on k+1 tokens in one batched forward, whose logits differ from single-token decode in the last bits, enough to flip a near-tie on this 2-bit model.

scratchy finds the head inside the checkpoint itself, with no per-model drafter configuration:

cargo build --release -p scratchy-cli --features metal,serve,model/qwen3.5-122b-a10b,spec/mtp,quant/mlx-affine-b4-g64-qembed
scr serve scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp
# log: MTP head from the target's checkpoint (46 tensors under language_model.mtp)

The head

  • MLX affine 4-bit, group size 64 on every linear layer, including the router and the shared-expert gate. The 256 routed experts are stacked as switch_mlp.
  • RMSNorm gains are stored in bf16 in MLX's full-gain form.
  • The head has no embeddings or lm_head of its own and uses the model's.
  • 4-bit is how MTP heads ship next to 2-bit targets. mlx-community's NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bit keeps its head at 4-bit/g64 too.

It is built from the official bf16 checkpoint by tools/mtp_drafter.py. That script downloads only the byte ranges of the 785 mtp.* tensors (4.7 GB), stacks the per-expert projections, adds 1 to the norm gains and quantizes with mx.quantize (mlx 0.32.2). tools/add_mtp.py then splices the result into the OptiQ files.

Validation

  • Recipe. tools/mtp_drafter.py, run on Qwen/Qwen3.6-35B-A3B, reproduces mlx-community/Qwen3.6-35B-A3B-MTP-4bit bit-exactly (46/46 tensors, identical config).
  • Expert order. Qwen3.5-122B stores one tensor per expert, which the 35B check does not cover.
    • Dequantized, each checked expert differs from its own bf16 source by 0.091–0.098 relative (4-bit noise). Against the next expert it differs by 1.15–1.92. That covers experts 0, 1, 137, 254 and 255 × gate/up/down (tools/xcheck_source.py).
    • It also agrees with the independently built 4-bit experts in scaryrawr/Qwen3.5-122B-A10B-oQ2.5e-mtp: 0.08–0.12, against ≥ 1.30 for the shifted control (tools/xcheck_experts.py).
  • Norms. All seven norm gains are bit-exact with mlx-community/Qwen3.5-122B-A10B-oQ4-mtp's inline head.
  • Loading. Read through this repo's own index and config, the head loads with strict=True into an mlx-lm-based MTP module, and a forward pass is finite.
  • Acceptance on this 2-bit model: 86.2%, 2.72 tokens a verify step, measured above. The head was trained on the full-precision model's hidden states; the 2-bit model shifts them, and the head still drafts well.

Offline acceptance oracle (mlx-lm, Mac with ≥ 64 GiB)

d=$(python -c "from huggingface_hub import snapshot_download as s; print(s('scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp', allow_patterns='tools/*'))")
bash "$d/tools/run-acceptance.sh"   # MLX_PY=<python with mlx-lm> to override ~/git/mlx-lm/.venv

The script greedy-generates three prompts, replays them teacher-forced, and reports draft tokens per verify step for k = 1..3. It does this over the full vocabulary and over the first 65536 ids. For comparison, mlx-community/Qwen3.6-35B-A3B-4bit with its MTP-4bit drafter, on the same prompts at k = 3 with the full vocabulary, gets 2.87 (code), 2.66 (story) and 3.54 (repeat).


The original model card of mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit follows, unchanged.

mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Read the write-up · All OptiQ quants · Docs

A 122-billion-parameter model that runs on a 36 GB Mac. This is a 2-bit mixed-precision MLX quant of Qwen3.5-122B-A10B (244 GB at bf16), produced by mlx-optiq. It is 44 GB on disk. While it generates, only ~12 GB sits in RAM: the attention, router and embeddings stay resident, and the 35 GB of mixture-of-experts weights stream off the SSD one expert at a time through optiq serve --stream-experts.

Image input

It takes images. The SigLIP vision tower rides at bf16 in a sidecar (optiq/optiq_vision.safetensors, 0.90 GB), so the image path keeps full precision while the language tower runs at 2 bits. The tower stays resident; only the experts stream.

Verified on a 36 GB M3 Max with expert streaming on, 11.6 GB resident:

image output
a red circle on white "The image contains a red circle."
a blue square on white "The image contains a blue square."
from PIL import Image
from optiq.runtime.moe_stream import load_streaming
from optiq.runtime.engine import OptiqEngine

model, tok = load_streaming("mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit")
eng = OptiqEngine.from_loaded(model, tok, "mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit")
print(eng.generate("What is in this image?", images=[Image.open("photo.jpg")]).text)

Requires mlx-optiq >= 0.3.3.

Asked to write Flappy Bird in a single HTML file, the 2-bit model produced a complete, working game. Here it is playing it:

Qwen3.5-122B-A10B 2-bit playing the Flappy Bird it wrote, on a 36 GB Mac

What it is

Property Value
Base Qwen3.5-122B-A10B (122 B total, ~10 B active per token, 256 experts/layer)
Method OptiQ static — structural per-layer bit allocation, no calibration
Bit-widths 4-bit on attention / router / embeddings / first+last block, 2-bit on the routed experts
Achieved bits-per-weight 2.50
On disk 44 GB
Resident while running ~12 GB (experts streamed)
Decode speed ~5 tok/s on an M3 Max (36 GB)

The allocation is rule-based: for a 122 B MoE, exact calibration-driven sensitivity would run for days and needs the full model resident as a reference, so OptiQ's static method assigns bits from architecture alone. On small models it matches the calibration method at a fraction of the cost (see the methods comparison).

Run it

This is a Qwen3.5 MoE (model_type: qwen3_5_moe), so it needs mlx-lm from main and import optiq (the MoE text tower postdates the 0.31.3 PyPI release; the main build also reports 0.31.3, so install from git, not a version pin):

pip install -U mlx-optiq "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git"

Serve it with SSD expert streaming (auto-enabled for a MoE too big to fit resident; --stream-experts forces it):

optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --stream-experts

Then open the Lab, ask for a game, and watch it render in the Canvas pane. Streaming keeps the residency flat (~12 GB) no matter how large the model on disk is, at the cost of per-token expert reads (decode is I/O-bound).

Notes

This is an extreme quant. 2-bit on the experts is lossy, and the point of this artifact is that a 122 B model runs at all on consumer Apple Silicon, with coherent output. For reference quality on this base, use a higher-bit quant (Qwen3.5-122B-A10B-4bit and up). The full story is in the blog post.

Downloads last month
55
Safetensors
Model size
125B params
Tensor type
U32
·
BF16
·
F32
·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp

Quantized
(180)
this model