TheCacheIsALie/RidgeFill-Projector-Qwen3.8-Flash-Next-i8

RidgeFill maps for Qwen3.8-Flash-Next: split 24 of 48, int8.

1.50 to 1.60× faster time to first token at 9K to 32K tokens in vLLM; RULER needles 99.8 against 100.0.

Data and code · All maps · llama.cpp fork · Citation

What this is

The per-layer maps for RidgeFill on Qwen3.8-Flash-Next, split at layer 24 of 48: 24 int8 maps for layers 24 to 47, a terminal-state correction for the 18 Gated DeltaNet layers above the split, and the token-frequency table that quality mode's protected rows use. Flash-Next uses hyper-connections, so each map reads all four residual streams entering layer 24 (10,240 values) and predicts one late layer's input (2,560 values). This is not a language model and runs only inside the serving engine below.

Files

ridgefill.json lists 27 files, 698,726,607 bytes (0.65 GiB), with their SHA256 hashes; its own SHA256 is 795363fa153f0e7e16454af6acabd644709e703d84de0dd9535565226bb51faa.

File(s)	Contents	Size
ridgefill.json	Manifest: credit, architecture fingerprints, fit metadata, and the SHA256 of every file	30,833 bytes
proj8.L24.safetensors … proj8.L47.safetensors (24 files)	Maps for layers 24 to 47. Each: proj.L.codes int8 [2560, 10240], proj.L.scale bf16 [2560, 80], proj.L.bias bf16 [2560]	26,629,560 bytes each
correction.safetensors	Gated DeltaNet terminal-state corrections st.L, f32 [48, 128, 128], for 18 layers	56,624,736 bytes
rowsel.safetensors	Token-frequency tables score, score_none, score_all, f32 [248320]	2,980,272 bytes
chat_template.jinja	Qwen's chat template with a RidgeFill header, listed in the manifest (not used by the paper's runs); Qwen Community License 1.0	12,159 bytes
source/kva-big-s24.safetensors	The bf16 fit these int8 maps were quantized from (the manifest's fit.proj_sha256). The Flash-Next run kit copies its final map into the engine's files; no reported number reads that map. Not needed to serve the maps	1,468,152,288 bytes
LICENSE, LICENSE-Qwen, NOTICE, CITATION.cff, SHA256SUMS	Licences, attribution, citation and checksums

The maps and the manifest are Apache-2.0. chat_template.jinja is Qwen's and stays under the Qwen Community License 1.0 (LICENSE-Qwen). Using the maps with Qwen3.8-Flash-Next is subject to that model's licence.

Use

The paper's runs served tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ (revision 6fe3e746) in our vLLM fork, shipped as the public image tcclaviger/vllm:29.08.10 (digest sha256:809657dd…). They used the run kit in the data repository's code/c5/, whose convert.py dequantizes these int8 maps to bf16 at load, and the determinism fix in code/engine-fix/. The fork's patches are not public; the image is.

hf download TheCacheIsALie/RidgeFill-Projector-Qwen3.8-Flash-Next-i8 --local-dir projector --exclude "source/*"

Add source/kva-big-s24.safetensors only to rebuild the run kit's engine files byte for byte.

Results

Two AMD Radeon AI PRO R9700. Exact is the same server with the fill off. Time to first token is measured at the client, as the ratio of medians of five timed requests at each length, alternating with exact on the same server start.

Arm	Perplexity gap on novel tokens [95 % CI]	RULER, all	TTFT 9K	TTFT 16K	TTFT 32K	64K request
Exact	n/a	100.0	1.00×	1.00×	1.00×	82.6 s
RidgeFill	+2.65 % [−0.02 %, +4.93 %]	99.8	1.50×	1.54×	1.60×	47.5 s (1.74×)
Quality mode	+1.60 % [−0.83 %, +4.32 %]	99.9	1.25×	1.22×	1.27×	55.6 s (1.49×)

Quality mode adds the terminal-state correction and protected-token rows. Its paired difference from plain RidgeFill is −1.02 % [−2.87 %, +0.80 %], so the improvement is not established, and it gives up about half of the speedup. About two thirds of the routed experts live in host memory and are read over PCIe, so this deployment cannot separate the compute the fill saves from the PCIe traffic it saves.

How the maps were fitted

On the served MXFP4/FP8 model's own activations, captured through the vLLM fork: 151 documents run whole, plus chat copies of 102 of them, with every eighth token position as a row (425,789 rows; the chat rows are weighted to half the total). The ridge penalty λ = 0.03 was chosen from six values between 0.001 and 0.3 by held-out block-input cosine over ten captures. The terminal-state correction was fitted on 13 further prompts. The int8 conversion stores each row as int8 codes with one bf16 scale per 128 weights.

How RidgeFill works

The lower layers run exactly up to a split. For every prompt row except the exact ones, a ridge map per later layer predicts that layer's input from the split layer's state, and the layer writes its cache from the prediction (b) instead of running in full: its own key and value projections run unchanged, and on a Gated DeltaNet layer so does its state update. The maps are fitted once, in closed form, from one pass over calibration text. The last few thousand prompt tokens, any selected exact rows and every decoding step run the full model. Nothing is trained and no token is dropped. The design comes from DeepSeek-V4.1-Flash, where the projection is trained into the model; (a) is kishida's Q3-8B-KVA-Projector, which retrofitted it onto Qwen3-8B with trained key and value heads.

Citation

The paper is RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill by Dylan Johnston and Robert P Smith (preprint, 2026, doi:10.5281/zenodo.23890247). Cite it as:

@misc{johnston2026ridgefill,
  author    = {Johnston, Dylan and Smith, Robert P},
  title     = {{RidgeFill}: Training-Free Late-Layer Cache Fill for Faster Prefill},
  year      = {2026},
  publisher = {Zenodo},
  note      = {Preprint},
  doi       = {10.5281/zenodo.23890247},
  url       = {https://doi.org/10.5281/zenodo.23890247}
}

To cite the data and code themselves, use the data release's citation (Approximate-Science/ridgefill-data).

An earlier technical note is archived at doi:10.5281/zenodo.23179168. The paper replaces some of its numbers; SUPERSEDED.md in the data repository lists which.