Instructions to use scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp") config = load_config("scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp
This is mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit
at 8284ab1029a2b0114824175aa7f04740ff318934 with the multi-token-prediction (MTP) head of
Qwen/Qwen3.5-122B-A10B put back. The OptiQ
conversion drops that head, and speculative decoding needs it.
The head is inline, as in the official checkpoint and
mlx-community/Qwen3.5-122B-A10B-oQ4-mtp.
Its 46 tensors are named language_model.mtp.*, live in model-mtp.safetensors (1.32 GiB), and are
listed in model.safetensors.index.json. Every quantized head module has its own
quantization/quantization_config entry. Total on disk is 44.1 GiB, against 42.8 GiB without the head.
Everything else is byte-identical to the OptiQ repo. That covers the nine weight shards, the vision
sidecar, the tokenizer, and every existing config, index and quantization entry. The copy was
sha256-checked file by file. mlx-lm's qwen3_5_moe sanitize drops language_model.mtp.*, so the model
loads and runs exactly as the OptiQ original does.
Speculative decoding with its head
M5 Max, scratchy (PR
#419), scr chat --bench, median of three
rounds alternated with plain decode:
| tok/s | TTFT | ITL p50 | |
|---|---|---|---|
| plain decode | 97.3 | 91.2 ms | 10.29 ms |
| drafting with this head (2 drafts a step) | 143.9 (+48%) | 94.8 ms | 18.67 ms |
The head drafts 2.72 tokens a verify step, 86.2% of drafts accepted (162 of 188; scr serve, one
greedy request, 256 tokens).
Greedy output with drafting matches plain decode until a single near-tie token, where it takes the other choice; each path is exactly reproducible. The verify step runs the target on k+1 tokens in one batched forward, whose logits differ from single-token decode in the last bits, enough to flip a near-tie on this 2-bit model.
scratchy finds the head inside the checkpoint itself, with no per-model drafter configuration:
cargo build --release -p scratchy-cli --features metal,serve,model/qwen3.5-122b-a10b,spec/mtp,quant/mlx-affine-b4-g64-qembed
scr serve scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp
# log: MTP head from the target's checkpoint (46 tensors under language_model.mtp)
The head
- MLX affine 4-bit, group size 64 on every linear layer, including the router and the shared-expert
gate. The 256 routed experts are stacked as
switch_mlp. - RMSNorm gains are stored in bf16 in MLX's full-gain form.
- The head has no embeddings or
lm_headof its own and uses the model's. - 4-bit is how MTP heads ship next to 2-bit targets. mlx-community's
NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bitkeeps its head at 4-bit/g64 too.
It is built from the official bf16 checkpoint by tools/mtp_drafter.py. That script downloads only the
byte ranges of the 785 mtp.* tensors (4.7 GB), stacks the per-expert projections, adds 1 to the norm
gains and quantizes with mx.quantize (mlx 0.32.2). tools/add_mtp.py then splices the result into the
OptiQ files.
Validation
- Recipe.
tools/mtp_drafter.py, run onQwen/Qwen3.6-35B-A3B, reproducesmlx-community/Qwen3.6-35B-A3B-MTP-4bitbit-exactly (46/46 tensors, identical config). - Expert order. Qwen3.5-122B stores one tensor per expert, which the 35B check does not cover.
- Dequantized, each checked expert differs from its own bf16 source by 0.091–0.098 relative (4-bit noise).
Against the next expert it differs by 1.15–1.92. That covers experts 0, 1, 137, 254 and 255 × gate/up/down
(
tools/xcheck_source.py). - It also agrees with the independently built 4-bit experts in
scaryrawr/Qwen3.5-122B-A10B-oQ2.5e-mtp: 0.08–0.12, against ≥ 1.30 for the shifted control (tools/xcheck_experts.py).
- Dequantized, each checked expert differs from its own bf16 source by 0.091–0.098 relative (4-bit noise).
Against the next expert it differs by 1.15–1.92. That covers experts 0, 1, 137, 254 and 255 × gate/up/down
(
- Norms. All seven norm gains are bit-exact with
mlx-community/Qwen3.5-122B-A10B-oQ4-mtp's inline head. - Loading. Read through this repo's own index and config, the head loads with
strict=Trueinto an mlx-lm-based MTP module, and a forward pass is finite. - Acceptance on this 2-bit model: 86.2%, 2.72 tokens a verify step, measured above. The head was trained on the full-precision model's hidden states; the 2-bit model shifts them, and the head still drafts well.
Offline acceptance oracle (mlx-lm, Mac with ≥ 64 GiB)
d=$(python -c "from huggingface_hub import snapshot_download as s; print(s('scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp', allow_patterns='tools/*'))")
bash "$d/tools/run-acceptance.sh" # MLX_PY=<python with mlx-lm> to override ~/git/mlx-lm/.venv
The script greedy-generates three prompts, replays them teacher-forced, and reports draft tokens per
verify step for k = 1..3. It does this over the full vocabulary and over the first 65536 ids. For
comparison, mlx-community/Qwen3.6-35B-A3B-4bit with its MTP-4bit drafter, on the same prompts at k = 3
with the full vocabulary, gets 2.87 (code), 2.66 (story) and 3.54 (repeat).
The original model card of mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit follows, unchanged.
mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit
Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Read the write-up · All OptiQ quants · Docs
A 122-billion-parameter model that runs on a 36 GB Mac. This is a 2-bit mixed-precision MLX quant of Qwen3.5-122B-A10B (244 GB at bf16), produced by mlx-optiq. It is 44 GB on disk. While it generates, only ~12 GB sits in RAM: the attention, router and embeddings stay resident, and the 35 GB of mixture-of-experts weights stream off the SSD one expert at a time through optiq serve --stream-experts.
Image input
It takes images. The SigLIP vision tower rides at bf16 in a sidecar (optiq/optiq_vision.safetensors, 0.90 GB), so the image path keeps full precision while the language tower runs at 2 bits. The tower stays resident; only the experts stream.
Verified on a 36 GB M3 Max with expert streaming on, 11.6 GB resident:
| image | output |
|---|---|
| a red circle on white | "The image contains a red circle." |
| a blue square on white | "The image contains a blue square." |
from PIL import Image
from optiq.runtime.moe_stream import load_streaming
from optiq.runtime.engine import OptiqEngine
model, tok = load_streaming("mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit")
eng = OptiqEngine.from_loaded(model, tok, "mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit")
print(eng.generate("What is in this image?", images=[Image.open("photo.jpg")]).text)
Requires mlx-optiq >= 0.3.3.
Asked to write Flappy Bird in a single HTML file, the 2-bit model produced a complete, working game. Here it is playing it:
What it is
| Property | Value |
|---|---|
| Base | Qwen3.5-122B-A10B (122 B total, ~10 B active per token, 256 experts/layer) |
| Method | OptiQ static — structural per-layer bit allocation, no calibration |
| Bit-widths | 4-bit on attention / router / embeddings / first+last block, 2-bit on the routed experts |
| Achieved bits-per-weight | 2.50 |
| On disk | 44 GB |
| Resident while running | ~12 GB (experts streamed) |
| Decode speed | ~5 tok/s on an M3 Max (36 GB) |
The allocation is rule-based: for a 122 B MoE, exact calibration-driven sensitivity would run for days and needs the full model resident as a reference, so OptiQ's static method assigns bits from architecture alone. On small models it matches the calibration method at a fraction of the cost (see the methods comparison).
Run it
This is a Qwen3.5 MoE (model_type: qwen3_5_moe), so it needs mlx-lm from main and import optiq (the MoE text tower postdates the 0.31.3 PyPI release; the main build also reports 0.31.3, so install from git, not a version pin):
pip install -U mlx-optiq "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git"
Serve it with SSD expert streaming (auto-enabled for a MoE too big to fit resident; --stream-experts forces it):
optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --stream-experts
Then open the Lab, ask for a game, and watch it render in the Canvas pane. Streaming keeps the residency flat (~12 GB) no matter how large the model on disk is, at the cost of per-token expert reads (decode is I/O-bound).
Notes
This is an extreme quant. 2-bit on the experts is lossy, and the point of this artifact is that a 122 B model runs at all on consumer Apple Silicon, with coherent output. For reference quality on this base, use a higher-bit quant (Qwen3.5-122B-A10B-4bit and up). The full story is in the blog post.
- Downloads last month
- 55
4-bit
Model tree for scratchery/Qwen3.5-122B-A10B-OptiQ-2bit-mtp
Base model
Qwen/Qwen3.5-122B-A10B