Qwen3.5 9B — avg3 (Q5_K_M)

A weight-space uniform average of three GGUF builds of Qwen3.5 9B, requantized to Q5_K_M. It is a derived artefact: no fine-tuning was performed. Its purpose is to sit in the middle of the source models in weight space — closer to all of them than any of them is to the others.

Model file: Qwen3.5-9B-avg3-Q5_K_M.gguf — 6,467,967,168 bytes (6.02 GiB), 427 tensors, GGUF v3, alignment 32.

SHA256 ff4a17728060979b2922727ac9daf01f5a54dc438fd4558267fc83cad0c7c5f8

Quick start

# llama.cpp (needs a build with qwen35 support — llama.cpp b11374 verified)
llama-cli -m Qwen3.5-9B-avg3-Q5_K_M.gguf -p "The capital of France is" -n 64 -t 4

# OpenAI-compatible server
llama-server -m Qwen3.5-9B-avg3-Q5_K_M.gguf -c 8192 -t 4

GGUF only — there is no safetensors/transformers version here. Load it with llama.cpp (or any GGUF-capable runtime). Architecture id in the file: qwen35.


What was merged

# Source file Arch Blocks Tensors Quant mix as shipped
1 MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf qwen35 32 427 F32×225, Q8_0×70, Q5_K×129, Q6_K×3
2 Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf qwen35 32 427 F32×177, Q5_K×217, Q6_K×33
3 Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf qwen35 32 427 F32×177, Q5_K×217, Q6_K×33

All three share identical tensor names, tensor order and tensor shapes, so a per-tensor mean is well defined. Declared bases: Qwen/Qwen3.5-9B (source 1) and unsloth/Qwen3.5-9B (sources 2 and 3). Source 2 upstream: Jackrong/Qwen3.5-9B-DeepSeek-V4-Flash.

Note that despite the identical _Q5_K_M filename suffix the three files are not quantized identically: source 1 keeps 70 tensors at Q8_0. That does not affect the merge, only the per-source dequantization step.

Deliberately excluded

Qwen3.8-9B-Q5_K_M.gguf was not merged. It has 33 blocks instead of 32 plus a multi-token-prediction head (qwen35.nextn_predict_layers = 1), i.e. 442 tensors including a blk.32.* tensor group that has no counterpart in the others. Averaging it would have required inventing tensors or silently dropping a whole layer, so it was left out.

Caveat: two of the three sources are near-duplicates

Sources 2 and 3 are not independent. A byte-level payload comparison gives:

  • 294 of 427 tensors byte-identical
  • only 2,834 differing bytes out of 6,457,001,984 (0.00004 %)

So the uniform 3-way average is effectively ⅓·MiMo + ⅔·DeepSeek-V4-Flash. Treat this model as "DeepSeek-V4-Flash with MiMo blended in", not as an average of three genuinely distinct checkpoints. Source 3's metadata carries a commit-style identifier (719b5158bed2e3f88829e1d1509a036fe7edac7d) in general.finetune, suggesting it is a re-quantization of the same weights.


How it was merged

For every tensor:

W = ( dequant(W1) + dequant(W2) + dequant(W3) ) / 3      # float32
out = quantize(W)                                          # target type per tensor
  • The quantized integers are never averaged directly. Every k-quant block carries its own d/dmin scales, so the stored integers of two models are not in a common code — averaging them is meaningless. Sources are dequantized to float32, averaged, then requantized.
  • llama.cpp's own quantizers were used, loaded via ctypes from libggml-base.so (dequantize_row_* / quantize_row_*_ref), so the block layouts are bit-exact with the reference implementation rather than a reimplementation of them.
  • The target quant mix is reused from source 2, a genuine llama-quantize Q5_K_M build: Q6_K for output.weight, ffn_down.weight, attn_qkv.weight and attn_v.weight, Q5_K elsewhere, F32 for 1-D tensors (norms, SSM parameters). general.file_type stays 17 (MOSTLY_Q5_K_M). No importance matrix was used — source 2 shipped without one either.
  • Large tensors are processed in row chunks. output.weight alone is 1.02 B parameters (4 GB in float32), so chunking keeps peak RSS at ~580 MB on a 4-core / 15 GB machine, with no intermediate F16 copy and a streaming write.
  • KV metadata (tokenizer, chat template, rope/SSM hyper-parameters) is inherited verbatim from source 2; only general.name, general.description and general.url were changed, and mradermacher.* attribution keys were removed. general.source.url still points at the upstream DeepSeek model, which is the correct provenance for a derived artefact.

Verification

1. Numerical exactness. For each of the 427 tensors, the float32 mean was re-derived independently from the sources, requantized with the target type, and compared against what the file actually stores:

checked 427 tensors | worst rmse(stored, ideal_requantized_mean) / sigma = 0.000e+00

i.e. every stored tensor is bit-identical to the ideal requantized average. The residual error against the exact float32 mean is purely the unavoidable Q5_K/Q6_K rounding: 0.00018–0.00051 rms, against a per-tensor sigma of 0.009–0.027 for the quantized weight matrices.

2. Structure. Tensor names, order, shapes, per-tensor quant types and byte sizes match the reference exactly; offsets are contiguous and data_start + Σpadded_sizes == filesize (6,467,967,168 bytes). Parsed successfully by both the llama.cpp loader and the official gguf-py reader (general.file_type = 17, 427 tensors).

3. Generation. Loads and generates coherent text under llama.cpp b11374.

4. Perplexity on one fixed ~1.66 k-token text (26 chunks of 64), scored with --ctx-size 512 (3 chunks) under identical conditions for all four models:

Model PPL
MiMo-V2.6-Distill-Qwen-9B 2.7639 ± 0.1759
Qwen3.5-9B-DeepSeek-V4-Flash 2.8784 ± 0.1876
Qwen3.5-9B-DeepSeek-V4-Flash V2 2.8727 ± 0.1867
Qwen3.5-9B-avg3 (this model) 2.7573 ± 0.1768

The merge is lowest of the four, but do not over-interpret this: on a sample this small the error bars are ±0.18 and the gap (≈0.11 against the best parent) is within noise. The honest claim is that merging did not degrade the model on this measurement.


Model details

Architecture qwen35 — hybrid linear-attention (SSM) + full attention
Parameters 8.954 B (427 tensors)
Layers 32, full attention every 4th layer (full_attention_interval = 4)
Embedding 4096
FFN 12288 (3 matrices per block)
Attention 16 heads / 4 KV heads, key & value length 256 (GQA)
SSM inner 4096, time-step rank 32, group count 16, conv kernel 4, state size 128
RoPE base 1e7, dimension count 64, sections [11, 11, 10, 0]
RMSNorm eps 1e-6
Context length 262144 (as declared in the GGUF)
Vocab 248320 tokens, 247587 merges, BPE (gpt2 tokenizer, qwen35 pretokenizer)
EOS / PAD 248046 / 248055
Languages (per parent metadata) en, zh, ko, ja, es, ru
Quantization Q5_K_M — 217×Q5_K, 33×Q6_K, 177×F32
File 6,467,967,168 bytes, alignment 32, GGUF v3

Intended use and limitations

  • Intended as a research / experimentation artefact: a cheap, reproducible way to obtain a single "central" checkpoint from several quantized builds of the same base model, e.g. to study model souping, to reduce the variance of a small sample of fine-tunes, or as a starting point for further merging or LoRA training.
  • Not instruction-tuned after merging. It inherits whatever behaviour the parents had; the average was not trained, so nothing guarantees improved instruction following, reasoning or factuality. Weight averaging can equally produce a model that regresses on some task while improving on none.
  • Provenance is partial. Source 1's GGUF carries no general.license field; the merged file declares apache-2.0 because that is what source 2 declared. Respect the upstream licenses of Qwen3.5-9B and of each fine-tune you derive from.
  • Perplexity above was measured on a tiny sample; it is a sanity check, not a benchmark.

Reproducing

Everything needed is in scripts/ in this repo (none of it is part of llama.cpp):

File Purpose
scripts/gguf_merge.py the merge itself: dequantize → average in float32 → requantize
scripts/verify_merge.py structural + bit-exactness verification of the result
scripts/compare_payloads.py byte-level tensor payload comparison between two GGUFs (how the near-duplicate sources were identified)
scripts/run_merge.sh crash/kill-resilient driver that re-invokes the merge with --resume
scripts/ppl_test.txt the perplexity fixture (the long variant used was this file repeated 4×)

Requirements: Python 3 + numpy only. No torch, no transformers, no gguf-py — the GGUF reader and writer in gguf_merge.py is self-contained, and all dequantization/quantization is delegated to llama.cpp's own C routines loaded with ctypes from libggml-base.so. Point the tool at your build with GGML_BASE_LIB=/path/to/libggml-base.so.0.25.3 (it also falls back to /root/llama-b11374/libggml-base.so.0.25.3, which is where it was developed).

# 1. merge (uniform weights over the three sources; --ref 1 = reuse source 2's
#    KV metadata and its Q5_K_M per-tensor quant assignment)
GGML_BASE_LIB=/path/to/libggml-base.so python scripts/gguf_merge.py \
  -o Qwen3.5-9B-avg3-Q5_K_M.gguf \
  -i MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf \
     Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
     Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf \
  --ref 1 --threads 4

# 2. verify: must report 427/427 tensors at rmse 0.000e+00
GGML_BASE_LIB=/path/to/libggml-base.so python scripts/verify_merge.py \
  --merged Qwen3.5-9B-avg3-Q5_K_M.gguf \
  --ref Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
  -i MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf \
     Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M.gguf \
     Qwen3.5-9B-DeepSeek-V4-Flash.Q5_K_M_V2.gguf

# 3. optional: how similar are two builds? (per-tensor byte diff)
python scripts/compare_payloads.py

Useful flags: --weights for non-uniform weighting, --ref to choose which source's quant mix to reuse, --dry-run to validate compatibility and print the target size without writing, --resume so a killed run costs at most one tensor, --keep-data to rewrite only the header of an existing output (refuses if data_start would move), --chunk-elems to trade RAM for throughput, and --kv-set KEY=VALUE to override metadata such as general.url.

The merge is deterministic, so a re-run reproduces this file byte for byte.

Downloads last month
50
GGUF
Model size
9B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support