---
license: apache-2.0
base_model:
- RecursiveSI/flint
- unsloth/flint-GGUF
base_model_relation: quantized
library_name: ninfer
pipeline_tag: text-generation
tags:
- ninfer
- gguf
- flint
- moe
- coding
- agent
- tool-calling
- reasoning
---

# Flint UD-Q4_K_XL, NInfer v3 artifact

A [NInfer](https://github.com/Neroued/ninfer) **v3** artifact of Recursive's
[Flint](https://huggingface.co/RecursiveSI/flint), a 39.8B-parameter mixture-of-experts model for
coding agents with about 2.5B parameters active per token, from Unsloth's
[UD-Q4_K_XL GGUF](https://huggingface.co/unsloth/flint-GGUF), for **NInfer-all**, the `master`
branch of [iamwavecut/ninfer-all](https://github.com/iamwavecut/ninfer-all). Every matrix keeps the
GGUF's ggml blocks byte for byte, so nothing is requantized; the engine multiplies the blocks in
place. Stock NInfer builds refuse this file: the Flint family and the `gguf_*` formats exist only in
that line, Flint from commit `25fda583` on. Use a build from `e1f2d961` on: earlier ones decode some
prompts to garbage (see [Quality](#quality)).

It runs the whole model: one to eight concurrent requests with prompt-prefix reuse, the model's
reasoning in its own channel, its GLM-4.7 tool calls returned as OpenAI tool calls, structured
output, typed decisions, speculative decoding by context lookup, and the routed experts on the GPU,
in pinned host memory or read from this file.

## What is inside

| component | representation |
|---|---|
| 49 blocks: 35 sliding-window (513 positions, RoPE), 14 full-attention (no positional encoding) | the GGUF's blocks: Q8_0, Q5_K in 7 of the 49 blocks; Q, K and V of a block one parent where they share a format |
| blocks 0 and 1: dense SwiGLU MLP (6144) | Q8_0, gate and up rows one parent |
| 47 MoE blocks: 256 routed experts, 6 per token, sigmoid scores renormalized | [gate; up] banks in IQ4_XS, Q5_K, Q4_K or IQ3_S, down banks in Q5_K, IQ4_XS, Q6_K or Q8_0, block by block as the GGUF chose them; each expert's gate rows followed by its up rows (2 × 512), down 2,048 rows per expert |
| shared expert (1024) of each MoE block | Q8_0, Q5_K or IQ4_XS, gate and up rows one parent |
| token table; untied head (150,144 tokens) | Q8_0 |
| router rows, norms, per-head QK norms, attention sinks | FP32 (the router's BF16 rows widened exactly) |
| tokenizer, chat template, generation config | [`RecursiveSI/flint`](https://huggingface.co/RecursiveSI/flint) at revision `2a247be0` |

One `.ninfer` file of 25,109,604,096 bytes (23.4 GiB; SHA-256 `601165e76d16a6e9abe46a0cbec007215b47936cc8085a463a872d9fc37d8d97`), next to its conversion report,
`SHA256SUMS` and `NOTICE`: 589 objects: 293 FP32, 159 Q8_0, 85 Q5_K, 43 IQ4_XS, 3 Q4_K, 1 Q6_K and 1 IQ3_S.

Conversion command, from the repository's tree at `25fda583`, with Unsloth's GGUF repository at
revision `20468edf` (`flint-UD-Q4_K_XL.gguf`, sha256 `63c2ecc4…`) and the Hugging Face text files at
`2a247be0`:

```bash
mkdir -p hf
for f in config.json tokenizer.json tokenizer_config.json chat_template.jinja generation_config.json; do
  curl -fsSL -o hf/$f https://huggingface.co/RecursiveSI/flint/resolve/main/$f
done
hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir .
python3 -m tools.convert --model hf --recipe flint_gguf --source gguf=flint-UD-Q4_K_XL.gguf \
  --name Flint-UD-Q4_K_XL --out flint-UD-Q4_K_XL-ninfer-v3.ninfer --device cpu
```

The converter takes about 35 seconds on a 64-CPU container. The published file was written by an earlier revision of the same recipe; the converter at `25fda583` writes the same bytes except the header's random artifact id.

## Running

Build NInfer-all for your GPU ([Linux](https://github.com/iamwavecut/ninfer-all/blob/master/docs/build-linux.md),
[Windows](https://github.com/iamwavecut/ninfer-all/blob/master/docs/build-windows.md)), then:

```bash
hf download WaveCut/Flint-UD-Q4_K_XL-NInfer-v3 flint-UD-Q4_K_XL-ninfer-v3.ninfer --local-dir models

# An OpenAI- and Anthropic-compatible server: four concurrent requests over 64K-token contexts.
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint \
  --max-context 65536 --max-concurrency 4

# Experts in pinned host memory, for a 16 or 24 GB card.
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint \
  --max-context 65536 --expert-residency host

# Speculative decoding by context lookup (Recursive publishes no draft model).
ninfer-serve models/flint-UD-Q4_K_XL-ninfer-v3.ninfer --model-id flint --lookup-ngram 4
```

- **Memory.** With every expert on the GPU: 23.4 GiB of weights, 0.7 GiB of workspace and, per
  sequence, 28 KiB of F16 keys and values per token (the 14 full-attention blocks) beside 53 MiB of
  sliding-window rings, about 25.5 GiB for one 32K-token sequence: a 32 or 48 GB card.
  `--expert-residency host` keeps 1.9 GiB of weights, two 624 MiB layer banks and a cache of the
  most routed experts on the GPU and pins 21.5 GiB of host memory; `disk` reads the experts from
  this file through the OS page cache and pins 137 MiB.
- **Reasoning.** The model opens `<think>` itself when it decides to reason; the reasoning arrives
  as `reasoning_content`. `chat_template_kwargs: {"enable_thinking": false}` (or
  `reasoning_effort: "none"`) closes an empty reasoning block first, and Flint answers at once.
- **Tool calls.** Flint's GLM-4.7 markup comes back as OpenAI tool calls, each argument read by
  its declared type; a forced `tool_choice` starts the call for the model.
- **Sampling defaults.** The release's: temperature 1.0, top-p 0.95, top-k 20.

Not available for this model: quantized `--kv-dtype` formats (the cache is F16), MTP and DFlash
drafting, rotation scaling, images, pipeline stages.

## Quality

No task benchmarks were run on this artifact. Its numerics were checked against two references:

- **An FP64 forward** of the model over the GGUF's exactly decoded blocks
  (`tools/reference/flint.py`): on the first 640 tokens of WikiText (past the 513-position window)
  NInfer's total NLL is 1,938.86 against 1,931.84 (+0.36%), the same with experts on the GPU, in
  host memory or read from disk.
- **llama.cpp b11845** on the same GGUF: perplexity over the five 4,096-token windows of 120 KB of
  WikiText both score alike is 15.270 against 15.195 (+0.49%), per window -0.012 to +0.022 nats of
  mean NLL.

Prompt chunks quantize activations to q8_1 as llama.cpp's matrix kernels do. Flint's routed
experts reach 32-value sums past F16's range there (89,234 in block 46 on WikiText), which turns a
chunk to NaN; NInfer scales those rows by exact powers of two before the product. Decode and
verification rows keep the same F16 sums; builds before `e1f2d961` left them unguarded, and a decode
step that reached such a sum turned its logits to NaN and the text to garbage. The rounding of
a prompt pass depends on its shape, so a typed decision's label probabilities move with the pass
that reads them: for one three-way choice, 0.819 / 0.017 / 0.164 in FP64, 0.841 / 0.021 / 0.138
through a chat and 0.787 / 0.025 / 0.188 as a read alone
([details](https://github.com/iamwavecut/ninfer-all/blob/master/docs/flint.md#rounding-with-the-pass)).

## Speed

`ninfer_bench` against llama.cpp b11804's `llama-bench` on the same GGUF (flash attention, F16 KV),
one sequence, three repetitions, in tok/s. The file fits whole only on a 32 GB card. On the 24 GB
cards NInfer keeps the experts in pinned host memory (`--expert-residency host`) with the 19.3 GiB
device cache it sizes itself, and llama.cpp keeps the experts of its first four blocks on the CPU
(`-ncmoe 4`, the fewest that run a 10,000-token prompt):

| Card | Engine | Prefill 2,048 | Prefill 10,000 | Decode 128 | Decode after 10,000 |
|---|---|---:|---:|---:|---:|
| RTX 5090 | NInfer | 15,670 | 14,656 | 347.7 | 314.6 |
| RTX 5090 | llama.cpp | 8,702 | 8,679 | 222.7 | 209.2 |
| RTX 4090 | NInfer, host experts | 12,423 | 11,466 | 210.2 | 188.0 |
| RTX 4090 | llama.cpp, `-ncmoe 4` | 4,868 | 4,725 | 165.7 | 149.4 |
| RTX 3090 | NInfer, host experts | 6,225 | 5,755 | 161.4 | 140.2 |
| RTX 3090 | NInfer, disk experts | 4,495 | 3,879 | 130.0 | 108.8 |
| RTX 3090 | llama.cpp, `-ncmoe 4` | 2,633 | 2,553 | 79.9 | 77.1 |

The RTX 5090 ran at 575 W beside two EPYC 9654 (CUDA 13.4), the RTX 4090 at 500 W beside a Ryzen 9
7950X (CUDA 13.4), the RTX 3090 at 370 W beside a Threadripper PRO 3955WX (CUDA 12.8). The disk row
reads through the page cache of a host with 503 GB of RAM.

Context lookup (`--lookup-ngram 4`) on the RTX 5090, a greedy rewrite of a code module (1,024
tokens, reasoning first): 335.0 -> 377.7 tok/s, 164 of 255 proposed tokens accepted; the output
equals plain decoding's.

## Credits and license

Model, tokenizer, chat template and generation config: Recursive, Flint, under the
[Apache License 2.0](https://huggingface.co/RecursiveSI/flint/blob/main/LICENSE). The UD-Q4_K_XL
quantization: Unsloth, under the Apache License 2.0. Attribution notices are collected in
[`NOTICE`](NOTICE). This repository only re-packs those weights into NInfer's container.