Instructions to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M # Run inference directly in the terminal: llama cli -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M # Run inference directly in the terminal: llama cli -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Use Docker
docker model run hf.co/ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ZeroWw/Qwen3.5-9B-avg3-Q5_K_M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ZeroWw/Qwen3.5-9B-avg3-Q5_K_M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
- Ollama
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with Ollama:
ollama run hf.co/ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
- Unsloth Desktop
- Pi
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with Docker Model Runner:
docker model run hf.co/ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
- Lemonade
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Run and chat with the model
lemonade run user.Qwen3.5-9B-avg3-Q5_K_M-Q5_K_M
List all available models
lemonade list
- Hermes Agent
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ZeroWw/Qwen3.5-9B-avg3-Q5_K_M with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "ZeroWw/Qwen3.5-9B-avg3-Q5_K_M:Q5_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.5 9B — avg3 (Q5_K_M)
A weight-space uniform average of three GGUF builds of Qwen3.5 9B, requantized to Q5_K_M.
It is a derived artefact: no fine-tuning was performed. Its purpose is to sit in the middle of
the source models in weight space — closer to all of them than any of them is to the others.
Model file: Qwen3.5-9B-avg3-Q5_K_M.gguf — 6,467,967,168 bytes (6.02 GiB), 427 tensors,
GGUF v3, alignment 32.
| SHA256 | ff4a17728060979b2922727ac9daf01f5a54dc438fd4558267fc83cad0c7c5f8 |
Quick start
# llama.cpp (needs a build with qwen35 support — llama.cpp b11374 verified)
llama-cli -m Qwen3.5-9B-avg3-Q5_K_M.gguf -p "The capital of France is" -n 64 -t 4
# OpenAI-compatible server
llama-server -m Qwen3.5-9B-avg3-Q5_K_M.gguf -c 8192 -t 4
GGUF only — there is no safetensors/transformers version here. Load it with llama.cpp
(or any GGUF-capable runtime). Architecture id in the file: qwen35.
What was merged
| # | Source file | Arch | Blocks | Tensors | Quant mix as shipped |
|---|---|---|---|---|---|
| 1 | MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf |
qwen35 |
32 | 427 | F32×225, Q8_0×70, Q5_K×129, Q6_K×3 |
| 2 | Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf |
qwen35 |
32 | 427 | F32×177, Q5_K×217, Q6_K×33 |
| 3 | Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf |
qwen35 |
32 | 427 | F32×177, Q5_K×217, Q6_K×33 |
All three share identical tensor names, tensor order and tensor shapes, so a per-tensor mean is well defined. Declared bases: Qwen/Qwen3.5-9B (source 1) and unsloth/Qwen3.5-9B (sources 2 and 3). Source 2 upstream: Jackrong/Qwen3.5-9B-DeepSeek-V4-Flash.
Note that despite the identical _Q5_K_M filename suffix the three files are not quantized
identically: source 1 keeps 70 tensors at Q8_0. That does not affect the merge, only the
per-source dequantization step.
Deliberately excluded
Qwen3.8-9B-Q5_K_M.gguf was not merged. It has 33 blocks instead of 32 plus a
multi-token-prediction head (qwen35.nextn_predict_layers = 1), i.e. 442 tensors including a
blk.32.* tensor group that has no counterpart in the others. Averaging it would have required
inventing tensors or silently dropping a whole layer, so it was left out.
Caveat: two of the three sources are near-duplicates
Sources 2 and 3 are not independent. A byte-level payload comparison gives:
- 294 of 427 tensors byte-identical
- only 2,834 differing bytes out of 6,457,001,984 (0.00004 %)
So the uniform 3-way average is effectively ⅓·MiMo + ⅔·DeepSeek-V4-Flash. Treat this model as
"DeepSeek-V4-Flash with MiMo blended in", not as an average of three genuinely distinct
checkpoints. Source 3's metadata carries a commit-style identifier
(719b5158bed2e3f88829e1d1509a036fe7edac7d) in general.finetune, suggesting it is a re-quantization
of the same weights.
How it was merged
For every tensor:
W = ( dequant(W1) + dequant(W2) + dequant(W3) ) / 3 # float32
out = quantize(W) # target type per tensor
- The quantized integers are never averaged directly. Every k-quant block carries its own
d/dminscales, so the stored integers of two models are not in a common code — averaging them is meaningless. Sources are dequantized to float32, averaged, then requantized. - llama.cpp's own quantizers were used, loaded via
ctypesfromlibggml-base.so(dequantize_row_*/quantize_row_*_ref), so the block layouts are bit-exact with the reference implementation rather than a reimplementation of them. - The target quant mix is reused from source 2, a genuine
llama-quantizeQ5_K_Mbuild:Q6_Kforoutput.weight,ffn_down.weight,attn_qkv.weightandattn_v.weight,Q5_Kelsewhere,F32for 1-D tensors (norms, SSM parameters).general.file_typestays17(MOSTLY_Q5_K_M). No importance matrix was used — source 2 shipped without one either. - Large tensors are processed in row chunks.
output.weightalone is 1.02 B parameters (4 GB in float32), so chunking keeps peak RSS at ~580 MB on a 4-core / 15 GB machine, with no intermediate F16 copy and a streaming write. - KV metadata (tokenizer, chat template, rope/SSM hyper-parameters) is inherited verbatim from
source 2; only
general.name,general.descriptionandgeneral.urlwere changed, andmradermacher.*attribution keys were removed.general.source.urlstill points at the upstream DeepSeek model, which is the correct provenance for a derived artefact.
Verification
1. Numerical exactness. For each of the 427 tensors, the float32 mean was re-derived independently from the sources, requantized with the target type, and compared against what the file actually stores:
checked 427 tensors | worst rmse(stored, ideal_requantized_mean) / sigma = 0.000e+00
i.e. every stored tensor is bit-identical to the ideal requantized average. The residual error against the exact float32 mean is purely the unavoidable Q5_K/Q6_K rounding: 0.00018–0.00051 rms, against a per-tensor sigma of 0.009–0.027 for the quantized weight matrices.
2. Structure. Tensor names, order, shapes, per-tensor quant types and byte sizes match the
reference exactly; offsets are contiguous and data_start + Σpadded_sizes == filesize
(6,467,967,168 bytes). Parsed successfully by both the llama.cpp loader and the official
gguf-py reader (general.file_type = 17, 427 tensors).
3. Generation. Loads and generates coherent text under llama.cpp b11374.
4. Perplexity on one fixed ~1.66 k-token text (26 chunks of 64), scored with
--ctx-size 512 (3 chunks) under identical conditions for all four models:
| Model | PPL |
|---|---|
| MiMo-V2.6-Distill-Qwen-9B | 2.7639 ± 0.1759 |
| Qwen3.5-9B-DeepSeek-V4-Flash | 2.8784 ± 0.1876 |
| Qwen3.5-9B-DeepSeek-V4-Flash V2 | 2.8727 ± 0.1867 |
| Qwen3.5-9B-avg3 (this model) | 2.7573 ± 0.1768 |
The merge is lowest of the four, but do not over-interpret this: on a sample this small the error bars are ±0.18 and the gap (≈0.11 against the best parent) is within noise. The honest claim is that merging did not degrade the model on this measurement.
Model details
| Architecture | qwen35 — hybrid linear-attention (SSM) + full attention |
| Parameters | 8.954 B (427 tensors) |
| Layers | 32, full attention every 4th layer (full_attention_interval = 4) |
| Embedding | 4096 |
| FFN | 12288 (3 matrices per block) |
| Attention | 16 heads / 4 KV heads, key & value length 256 (GQA) |
| SSM | inner 4096, time-step rank 32, group count 16, conv kernel 4, state size 128 |
| RoPE | base 1e7, dimension count 64, sections [11, 11, 10, 0] |
| RMSNorm eps | 1e-6 |
| Context length | 262144 (as declared in the GGUF) |
| Vocab | 248320 tokens, 247587 merges, BPE (gpt2 tokenizer, qwen35 pretokenizer) |
| EOS / PAD | 248046 / 248055 |
| Languages (per parent metadata) | en, zh, ko, ja, es, ru |
| Quantization | Q5_K_M — 217×Q5_K, 33×Q6_K, 177×F32 |
| File | 6,467,967,168 bytes, alignment 32, GGUF v3 |
Intended use and limitations
- Intended as a research / experimentation artefact: a cheap, reproducible way to obtain a single "central" checkpoint from several quantized builds of the same base model, e.g. to study model souping, to reduce the variance of a small sample of fine-tunes, or as a starting point for further merging or LoRA training.
- Not instruction-tuned after merging. It inherits whatever behaviour the parents had; the average was not trained, so nothing guarantees improved instruction following, reasoning or factuality. Weight averaging can equally produce a model that regresses on some task while improving on none.
- Provenance is partial. Source 1's GGUF carries no
general.licensefield; the merged file declaresapache-2.0because that is what source 2 declared. Respect the upstream licenses of Qwen3.5-9B and of each fine-tune you derive from. - Perplexity above was measured on a tiny sample; it is a sanity check, not a benchmark.
Reproducing
Everything needed is in scripts/ in this repo (none of it is part of llama.cpp):
| File | Purpose |
|---|---|
scripts/gguf_merge.py |
the merge itself: dequantize → average in float32 → requantize |
scripts/verify_merge.py |
structural + bit-exactness verification of the result |
scripts/compare_payloads.py |
byte-level tensor payload comparison between two GGUFs (how the near-duplicate sources were identified) |
scripts/run_merge.sh |
crash/kill-resilient driver that re-invokes the merge with --resume |
scripts/ppl_test.txt |
the perplexity fixture (the long variant used was this file repeated 4×) |
Requirements: Python 3 + numpy only. No torch, no transformers, no gguf-py — the GGUF reader
and writer in gguf_merge.py is self-contained, and all dequantization/quantization is delegated
to llama.cpp's own C routines loaded with ctypes from libggml-base.so. Point the tool at your
build with GGML_BASE_LIB=/path/to/libggml-base.so.0.25.3 (it also falls back to
/root/llama-b11374/libggml-base.so.0.25.3, which is where it was developed).
# 1. merge (uniform weights over the three sources; --ref 1 = reuse source 2's
# KV metadata and its Q5_K_M per-tensor quant assignment)
GGML_BASE_LIB=/path/to/libggml-base.so python scripts/gguf_merge.py \
-o Qwen3.5-9B-avg3-Q5_K_M.gguf \
-i MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf \
Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf \
--ref 1 --threads 4
# 2. verify: must report 427/427 tensors at rmse 0.000e+00
GGML_BASE_LIB=/path/to/libggml-base.so python scripts/verify_merge.py \
--merged Qwen3.5-9B-avg3-Q5_K_M.gguf \
--ref Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
-i MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf \
Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf
# 3. optional: how similar are two builds? (per-tensor byte diff)
python scripts/compare_payloads.py
Useful flags: --weights for non-uniform weighting, --ref to choose which source's quant mix to
reuse, --dry-run to validate compatibility and print the target size without writing,
--resume so a killed run costs at most one tensor, --keep-data to rewrite only the header of an
existing output (refuses if data_start would move), --chunk-elems to trade RAM for throughput,
and --kv-set KEY=VALUE to override metadata such as general.url.
The merge is deterministic, so a re-run reproduces this file byte for byte.
- Downloads last month
- 50
5-bit