WaveCut/Flint-UD-Q4_K_XL-NInfer-v3

Flint UD-Q4_K_XL, NInfer v3 artifact

A NInfer v3 artifact of Recursive's Flint, a 39.8B-parameter mixture-of-experts model for coding agents with about 2.5B parameters active per token, from Unsloth's UD-Q4_K_XL GGUF, for NInfer-all, the master branch of iamwavecut/ninfer-all. Every matrix keeps the GGUF's ggml blocks byte for byte, so nothing is requantized; the engine multiplies the blocks in place. Stock NInfer builds refuse this file: the Flint family and the gguf_* formats exist only in that line, Flint from commit 25fda583 on. Use a build from e1f2d961 on: earlier ones decode some prompts to garbage (see Quality).

It runs the whole model: one to eight concurrent requests with prompt-prefix reuse, the model's reasoning in its own channel, its GLM-4.7 tool calls returned as OpenAI tool calls, structured output, typed decisions, speculative decoding by context lookup, and the routed experts on the GPU, in pinned host memory or read from this file.

What is inside

component	representation
49 blocks: 35 sliding-window (513 positions, RoPE), 14 full-attention (no positional encoding)	the GGUF's blocks: Q8_0, Q5_K in 7 of the 49 blocks; Q, K and V of a block one parent where they share a format
blocks 0 and 1: dense SwiGLU MLP (6144)	Q8_0, gate and up rows one parent
47 MoE blocks: 256 routed experts, 6 per token, sigmoid scores renormalized	[gate; up] banks in IQ4_XS, Q5_K, Q4_K or IQ3_S, down banks in Q5_K, IQ4_XS, Q6_K or Q8_0, block by block as the GGUF chose them; each expert's gate rows followed by its up rows (2 × 512), down 2,048 rows per expert
shared expert (1024) of each MoE block	Q8_0, Q5_K or IQ4_XS, gate and up rows one parent
token table; untied head (150,144 tokens)	Q8_0
router rows, norms, per-head QK norms, attention sinks	FP32 (the router's BF16 rows widened exactly)
tokenizer, chat template, generation config	RecursiveSI/flint at revision 2a247be0

One .ninfer file of 25,109,604,096 bytes (23.4 GiB; SHA-256 601165e76d16a6e9abe46a0cbec007215b47936cc8085a463a872d9fc37d8d97), next to its conversion report, SHA256SUMS and NOTICE: 589 objects: 293 FP32, 159 Q8_0, 85 Q5_K, 43 IQ4_XS, 3 Q4_K, 1 Q6_K and 1 IQ3_S.

Conversion command, from the repository's tree at 25fda583, with Unsloth's GGUF repository at revision 20468edf (flint-UD-Q4_K_XL.gguf, sha256 63c2ecc4…) and the Hugging Face text files at 2a247be0:

mkdir -p hf
for f in config.json tokenizer.json tokenizer_config.json chat_template.jinja generation_config.json; do
  curl -fsSL -o hf/$f https://huggingface.co/RecursiveSI/flint/resolve/main/$f
done
hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir .
python3 -m tools.convert --model hf --recipe flint_gguf --source gguf=flint-UD-Q4_K_XL.gguf \
  --name Flint-UD-Q4_K_XL --out flint-UD-Q4_K_XL-ninfer-v3.ninfer --device cpu

The converter takes about 35 seconds on a 64-CPU container. The published file was written by an earlier revision of the same recipe; the converter at 25fda583 writes the same bytes except the header's random artifact id.

Running

Build NInfer-all for your GPU (Linux, Windows), then:

hf download WaveCut/Flint-UD-Q4_K_XL-NInfer-v3 flint-UD-Q4_K_XL-ninfer-v3.ninfer --local-dir models

# An OpenAI- and Anthropic-compatible server: four concurrent requests over 64K-token contexts.
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint \
  --max-context 65536 --max-concurrency 4

# Experts in pinned host memory, for a 16 or 24 GB card.
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint \
  --max-context 65536 --expert-residency host

# Speculative decoding by context lookup (Recursive publishes no draft model).
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint --lookup-ngram 4

• Memory. With every expert on the GPU: 23.4 GiB of weights, 0.7 GiB of workspace and, per sequence, 28 KiB of F16 keys and values per token (the 14 full-attention blocks) beside 53 MiB of sliding-window rings, about 25.5 GiB for one 32K-token sequence: a 32 or 48 GB card. --expert-residency host keeps 1.9 GiB of weights, two 624 MiB layer banks and a cache of the most routed experts on the GPU and pins 21.5 GiB of host memory; disk reads the experts from this file through the OS page cache and pins 137 MiB.
• Reasoning. The model opens <think> itself when it decides to reason; the reasoning arrives as reasoning_content. chat_template_kwargs: {"enable_thinking": false} (or reasoning_effort: "none") closes an empty reasoning block first, and Flint answers at once.
• Tool calls. Flint's GLM-4.7 markup comes back as OpenAI tool calls, each argument read by its declared type; a forced tool_choice starts the call for the model.
• Sampling defaults. The release's: temperature 1.0, top-p 0.95, top-k 20.

Not available for this model: quantized --kv-dtype formats (the cache is F16), MTP and DFlash drafting, rotation scaling, images, pipeline stages.

Quality

No task benchmarks were run on this artifact. Its numerics were checked against two references:

• An FP64 forward of the model over the GGUF's exactly decoded blocks (tools/reference/flint.py): on the first 640 tokens of WikiText (past the 513-position window) NInfer's total NLL is 1,938.86 against 1,931.84 (+0.36%), the same with experts on the GPU, in host memory or read from disk.
• llama.cpp b11845 on the same GGUF: perplexity over the five 4,096-token windows of 120 KB of WikiText both score alike is 15.270 against 15.195 (+0.49%), per window -0.012 to +0.022 nats of mean NLL.

Prompt chunks quantize activations to q8_1 as llama.cpp's matrix kernels do. Flint's routed experts reach 32-value sums past F16's range there (89,234 in block 46 on WikiText), which turns a chunk to NaN; NInfer scales those rows by exact powers of two before the product. Decode and verification rows keep the same F16 sums; builds before e1f2d961 left them unguarded, and a decode step that reached such a sum turned its logits to NaN and the text to garbage. The rounding of a prompt pass depends on its shape, so a typed decision's label probabilities move with the pass that reads them: for one three-way choice, 0.819 / 0.017 / 0.164 in FP64, 0.841 / 0.021 / 0.138 through a chat and 0.787 / 0.025 / 0.188 as a read alone (details).

Speed

ninfer_bench against llama.cpp b11804's llama-bench on the same GGUF (flash attention, F16 KV), one sequence, three repetitions, in tok/s. The file fits whole only on a 32 GB card. On the 24 GB cards NInfer keeps the experts in pinned host memory (--expert-residency host) with the 19.3 GiB device cache it sizes itself, and llama.cpp keeps the experts of its first four blocks on the CPU (-ncmoe 4, the fewest that run a 10,000-token prompt):

Card	Engine	Prefill 2,048	Prefill 10,000	Decode 128	Decode after 10,000
RTX 5090	NInfer	15,670	14,656	347.7	314.6
RTX 5090	llama.cpp	8,702	8,679	222.7	209.2
RTX 4090	NInfer, host experts	12,423	11,466	210.2	188.0
RTX 4090	llama.cpp, -ncmoe 4	4,868	4,725	165.7	149.4
RTX 3090	NInfer, host experts	6,225	5,755	161.4	140.2
RTX 3090	NInfer, disk experts	4,495	3,879	130.0	108.8
RTX 3090	llama.cpp, -ncmoe 4	2,633	2,553	79.9	77.1

The RTX 5090 ran at 575 W beside two EPYC 9654 (CUDA 13.4), the RTX 4090 at 500 W beside a Ryzen 9 7950X (CUDA 13.4), the RTX 3090 at 370 W beside a Threadripper PRO 3955WX (CUDA 12.8). The disk row reads through the page cache of a host with 503 GB of RAM.

Context lookup (--lookup-ngram 4) on the RTX 5090, a greedy rewrite of a code module (1,024 tokens, reasoning first): 335.0 -> 377.7 tok/s, 164 of 255 proposed tokens accepted; the output equals plain decoding's.

Credits and license

Model, tokenizer, chat template and generation config: Recursive, Flint, under the Apache License 2.0. The UD-Q4_K_XL quantization: Unsloth, under the Apache License 2.0. Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.