Speed on the RTX 3090, 4090 and 5090; decode fix

README.md

@@ -26,7 +26,8 @@ coding agents with about 2.5B parameters active per token, from Unsloth's
 branch of [iamwavecut/ninfer-all](https://github.com/iamwavecut/ninfer-all). Every matrix keeps the
 GGUF's ggml blocks byte for byte, so nothing is requantized; the engine multiplies the blocks in
 place. Stock NInfer builds refuse this file: the Flint family and the `gguf_*` formats exist only in
-that line, Flint from commit `25fda583` on.
+that line, Flint from commit `25fda583` on. Use a build from `e1f2d961` on: earlier ones decode some
+prompts to garbage (see [Quality](#quality)).
 
 It runs the whole model: one to eight concurrent requests with prompt-prefix reuse, the model's
 reasoning in its own channel, its GLM-4.7 tool calls returned as OpenAI tool calls, structured
@@ -114,7 +115,9 @@ No task benchmarks were run on this artifact. Its numerics were checked against
 
 Prompt chunks quantize activations to q8_1 as llama.cpp's matrix kernels do. Flint's routed
 experts reach 32-value sums past F16's range there (89,234 in block 46 on WikiText), which turns a
-chunk to NaN; NInfer scales those rows by exact powers of two before the product. The rounding of
+chunk to NaN; NInfer scales those rows by exact powers of two before the product. Decode and
+verification rows keep the same F16 sums; builds before `e1f2d961` left them unguarded, and a decode
+step that reached such a sum turned its logits to NaN and the text to garbage. The rounding of
 a prompt pass depends on its shape, so a typed decision's label probabilities move with the pass
 that reads them: for one three-way choice, 0.819 / 0.017 / 0.164 in FP64, 0.841 / 0.021 / 0.138
 through a chat and 0.787 / 0.025 / 0.188 as a read alone
@@ -122,35 +125,29 @@ through a chat and 0.787 / 0.025 / 0.188 as a read alone
 
 ## Speed
 
-NVIDIA A40 (48 GB, 300 W, CUDA 12.8), one sequence, every expert on the GPU; `ninfer_bench` against
-llama.cpp b11845's `llama-bench` on the same GGUF with flash attention:
-
-| Test | NInfer tok/s | llama.cpp tok/s | NInfer / llama.cpp |
-|---|---:|---:|---:|
-| Prefill, 2,048 tokens | 5,685 | 3,295 | 1.73 |
-| Prefill, 10,000 tokens | 5,332 | 3,145 | 1.70 |
-| Prefill, 40,000 tokens | 4,399 | 2,759 | 1.59 |
-| Decode, 128 tokens | 149.6 | 114.3 | 1.31 |
-| Decode after 10,000 tokens | 135.0 | 106.9 | 1.26 |
-| Decode after 40,000 tokens | 112.3 | 92.2 | 1.22 |
-
-Context lookup (`--lookup-ngram 4`) on a greedy rewrite of a code module: 145.2 -> 227.9 tok/s for a
-direct 669-token answer, 145.1 -> 175.5 tok/s for a 1,024-token answer that reasons first; the
-output equals plain decoding's.
-
-Experts off the GPU on the same A40, with caches of 8 and 16 GiB standing for the room a 16 and a
-24 GB card leave (30 CPU threads of an EPYC 7662 take a share of the decode misses):
-
-| Experts | Cache | Prefill 2,048 | Decode 128 | Prefill 10,000 | Decode after it |
-|---|---:|---:|---:|---:|---:|
-| host | 8 GiB | 2,640 | 26.9 | 2,408 | 23.0 |
-| host | 16 GiB | 4,645 | 34.8 | 3,403 | 32.7 |
-| disk | 8 GiB | 286 | 10.0 | 321 | 7.2 |
-| disk | 16 GiB | 466 | 30.1 ± 20.9 | 592 | 13.2 |
-
-llama.cpp with its experts on the CPU (`-ot exps=CPU`) on the same host: 255.5 tok/s prefill and
-14.0 tok/s decode. The disk rows read through the page cache of a host with 2 TB of RAM in whatever
-state earlier runs left it.
+`ninfer_bench` against llama.cpp b11804's `llama-bench` on the same GGUF (flash attention, F16 KV),
+one sequence, three repetitions, in tok/s. The file fits whole only on a 32 GB card. On the 24 GB
+cards NInfer keeps the experts in pinned host memory (`--expert-residency host`) with the 19.3 GiB
+device cache it sizes itself, and llama.cpp keeps the experts of its first four blocks on the CPU
+(`-ncmoe 4`, the fewest that run a 10,000-token prompt):
+
+| Card | Engine | Prefill 2,048 | Prefill 10,000 | Decode 128 | Decode after 10,000 |
+|---|---|---:|---:|---:|---:|
+| RTX 5090 | NInfer | 15,670 | 14,656 | 347.7 | 314.6 |
+| RTX 5090 | llama.cpp | 8,702 | 8,679 | 222.7 | 209.2 |
+| RTX 4090 | NInfer, host experts | 12,423 | 11,466 | 210.2 | 188.0 |
+| RTX 4090 | llama.cpp, `-ncmoe 4` | 4,868 | 4,725 | 165.7 | 149.4 |
+| RTX 3090 | NInfer, host experts | 6,225 | 5,755 | 161.4 | 140.2 |
+| RTX 3090 | NInfer, disk experts | 4,495 | 3,879 | 130.0 | 108.8 |
+| RTX 3090 | llama.cpp, `-ncmoe 4` | 2,633 | 2,553 | 79.9 | 77.1 |
+
+The RTX 5090 ran at 575 W beside two EPYC 9654 (CUDA 13.4), the RTX 4090 at 500 W beside a Ryzen 9
+7950X (CUDA 13.4), the RTX 3090 at 370 W beside a Threadripper PRO 3955WX (CUDA 12.8). The disk row
+reads through the page cache of a host with 503 GB of RAM.
+
+Context lookup (`--lookup-ngram 4`) on the RTX 5090, a greedy rewrite of a code module (1,024
+tokens, reasoning first): 335.0 -> 377.7 tok/s, 164 of 255 proposed tokens accepted; the output
+equals plain decoding's.
 
 ## Credits and license