doth4580/Naive-N0.5-Flash-EXL3-4bpw

Naive-N0.5-Flash — EXL3 4 bpw (experts 4 / attention 6)

Original model: NaiveAI/Naive-N0.5-Flash by the NaiveAI Team, MIT License. This artifact is quantized from the bf16 master release (not the FP8 one). It contains modified weights (EXL3 trellis quantization); the original model is © its authors. Tokenizer and chat template are unchanged from the original. Served by the veloGB10 engine it is listed by /v1/models as doth4580/Naive-N0.5-Flash-EXL3-4bpw (the model_name: line of this card); base_model: is the original model it was quantized from, not this pack.

A sibling of doth4580/Naive-N0.5-Flash-EXL3-3.5bpw: an EXL3 (ExLlamaV3 trellis) mixed-bitrate quantization of Naive-N0.5-Flash — 309B parameters, 15.5B active, native 1M-token context — with the routed experts at 4 bpw and attention, the DSA indexer, the dense layer-0 MLP and lm_head at 6 bpw (4.03 bpw overall). The pack is 157 GB (147 GiB) against ~315 GB for the FP8 release and ~618 GB for bf16; it fits across two NVIDIA DGX Spark (GB10) machines with less headroom than the 3.5 bpw sibling, or across four comfortably.

Quantized with stock exllamav3 1.5.3 plus the architecture definition from sf-stav/Naive-N0.5-Flash-EXL3-2x-DGX-Spark, run on 8× RTX PRO 6000 (Blackwell) in about 1.5 hours, with a 498-row × 2,048-token calibration set built from code repositories, agent-style tool-calling conversations rendered through the model's own chat template, and general text (instead of the converter's default 250-row general set).

Accuracy

Forward-only gate in exllamav3 (teacher-forced NLL on a 977-token technical text; next-token probability after two code/JSON prefixes), same prompts for every row:

model	source	NLL (nats/token)	p(8-space indent after if n < 0:)	p(age) on a JSON key
Naive-N0.5-Flash bf16, transformers	bf16	1.627	1.000	1.000
this pack (4.03 bpw)	bf16	1.634	1.000	1.000
sibling 3.5 bpw pack	FP8	1.619	1.000	1.000

So this pack is within 0.007 nats of its own unquantized source. It is not better than the 3.5 bpw sibling on that text: the FP8 release and the bf16 master are different checkpoints (their embedding tables differ by ~14 % relative), and on this particular text the FP8-derived pack scores marginally lower. Agentic tool use with tool-eval-bench v2.7.1 (69 scenarios, temperature 0, seed 42, thinking on), served by the veloGB10 engine at TP=4: sibling 3.5 bpw pack 86 / 100 (118 / 138 points); this pack 91 / 100 (126 / 138). One run each; the bench itself warns a single trial is not a calibrated difficulty estimate, so read it as "at least as good", not as a five-point lead. Where this pack may pay is code and tool-call output, which its calibration set targets, and in the headroom that 4-bit experts leave against outliers; measure on your own workload. Note that a pack built by mixing the two sources (attention from bf16, experts copied from the FP8-derived pack) scored 2.48 nats — the two releases do not mix.

Serving

• ExLlamaV3 directly: upstream exllamav3 ≥ 1.5.3 with the architecture definition from the recipe repository applied. The pack carries the native exllamav3 metadata (config.json with its quantization_config, quantization_config.json, model.safetensors.index.json).
• veloGB10 (sf-stav/veloGB10), the from-scratch GB10 engine this pack was made for: it loads per-tensor bit rates (3 / 3.5 / 4 / 5 / 6 bpw) from the first release that includes the Naive-N0.5 mixed-rate loader.
• The vLLM + vllm-exl3 recipe used by the 3.5 bpw sibling has not been tested with this pack. That stack was validated for a uniform 3.5 bpw pack; a mixed-rate pack may need plugin changes. The vLLM-rewritten config.json / .native pair of the sibling is therefore not shipped here.

Important for any runtime: layer 47's routed up_proj is stored divided by 128 (exllamav3's interm_div for this architecture — that layer's expert activations overflow f16) and the forward must scale that layer's routing weights by 128, as exllamav3 does. A runtime that applies the pack as stored silently produces a last-layer MoE 128× too small (the symptom is loops and wrong token choices, not a crash).

Specifications

Base model	NaiveAI/Naive-N0.5-Flash (bf16 master as the direct source)
Architecture	MoE, 309B total / 15.5B active, 48 layers, hidden 4096, 64 heads / 4 KV heads, head_dim 192
Attention	hybrid 39 SWA layers (window 128) + 9 DSA layers (top-2048 selection), GQA4, no full attention
MoE	256 routed experts, 8 per token, expert intermediate 2048
Context / vocab	1,048,576 tokens / 152,576
Quantization	EXL3 trellis, mul1 codebook — routed experts 4 bpw; attention, DSA indexer, dense MLP and lm_head 6 bpw (4.03 bpw overall); embeddings unquantized; calibration 498×2048 (code / agent / general)
Format / size	native exl3 — 157 GB (147 GiB), 24 shards
License	MIT (inherited from the original model)

Files

File	Notes
model-00001..00024-of-00024.safetensors	the quantized weights
model.safetensors.index.json, quantization_config.json	tensor index and the full EXL3 metadata
config.json	the native ExLlamaV3 pack config (carries quantization_config)
RECIPE.yaml	the per-tensor bit-rate recipe the converter was run with
GATE_exllamav3.json, CONVERT.log	the gate numbers above and the converter's full log
SHA256SUMS	checksums of every file in the pack
modeling_naive_n05_flash.py, configuration_naive_n05_flash.py	model code, loaded via trust_remote_code
tokenizer + chat_template.jinja, generation_config.json	unchanged from the original

Notes

• Beyond 2,048 tokens the DSA layers are approximate in runtimes that run them as dense attention (exact up to 2,048 tokens); the 1M native context needs a sparse attention backend.
• Larger than the sibling: ~14 % more bytes than the 3.5 bpw pack, so expect proportionally lower decode speed on bandwidth-bound hardware such as the DGX Spark.
• Sampling defaults from the original generation_config.json: temperature 1.0, top_p 0.95.

License and credits

• Original model: NaiveAI/Naive-N0.5-Flash — © the NaiveAI Team, MIT; a copy ships here as LICENSE. Built on Xiaomi's MiMo-V2.5 and DeepSeek Sparse Attention.
• This artifact: EXL3 mixed-bitrate quantization produced with exllamav3 1.5.3 and the architecture definition from the recipe repository. Weights are modified; tokenizer and chat template are not.
• Sibling: doth4580/Naive-N0.5-Flash-EXL3-3.5bpw (uniform 3.5 bpw from the FP8 release, with the vLLM recipe).