K2-Horizon-MoVA-36B-A4B FAST2 πŸš€

3.1Γ— faster decode at 61K context. 2.1Γ— faster prefill. Weights bit-identical to APEX-Mini. Zero perplexity loss inside the window.

This is a recipe repo β€” it deliberately contains no model weights. FAST2 is not a different file: it is the original VladHong/K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf plus a 160-byte metadata key (an 8192-token sliding-window attention declaration) and the companion llama.cpp fork that honors it. Build it yourself in under a minute, below.

The numbers (measured, single session, 61,337-token prompt, 5 interleaved rounds)

@ 61K context vanilla APEX-Mini* FAST2 speedup
decode tok/s 16.48 50.72 3.07Γ—
prefill tok/s 692 1,482 2.14Γ—
@ 32K context vanilla FAST2 speedup
decode tok/s 36.78 52.64 1.43Γ—

At short context FAST2 matches vanilla (~59 tok/s β€” weight-bandwidth bound, untouched by design). Raw aggregates and spreads (0.4–2.1%): results-head2head.json.

* Vanilla with its default f16 KV cache cannot serve 61K context on a 22 GB GPU at all (OOM); the comparator is vanilla with q8_0 KV β€” the only configuration in which vanilla fits.

Get it running (two steps)

1. Get the base GGUF β€” VladHong/K2-Horizon-MoVA-36B-A4B-APEX-Mini-GGUF

2. Get the patched llama.cpp β€” VladHong/llama-cpp-K2-FAST2 (complete source snapshot; the same code is branch k2-fast2 on ModelScope). Build it exactly like the MBZUAI fork, plus -DGGML_CUDA_FA_ALL_QUANTS=ON:

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
      -DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

Then either (a) window at load time β€” no file changes at all:

llama-server -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf -c 65536 -ngl 99 --parallel 1 \
    -fa on -ctk q4_0 -ctv f16 -b 2048 -ub 2048 -t 8 \
    --override-kv k2-horizon.attention.sliding_window=int:8192 --port 8099

or (b) bake the window into a self-contained GGUF (~12 seconds, from this repo's make_swa_gguf.py):

python3 make_swa_gguf.py K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf 8192 K2-FAST2-SWA8K.gguf
llama-server -m K2-FAST2-SWA8K.gguf -c 65536 -ngl 99 --parallel 1 -fa on -b 2048 -ub 2048 -t 8 --port 8099

fast2-serve.sh wraps all of this with VRAM gating and context-aware flags (FAST2_SWA=8192 to enable the window through the launcher).

Quality: measured, not hand-waved

  • Perplexity inside the window is numerically identical to full attention (2.1814 vs 2.1814 on a 240K-token stratified corpus; Ξ” 0.0006 at 16K context with W=16384).
  • 61K-context needle retrieval: PASS inside the window β€” exact answer; outside it the model declines instead of hallucinating (clean failure mode).
  • The tradeoff is explicit: attention recall covers the last W tokens (8192 here; rebuild with W=16384 β†’ 43.90 tok/s @61K, still 2.7Γ— vanilla, recall 16K).

Why it's fast (one paragraph)

K2-Horizon decodes slowly at long context because all 48 layers run full attention over the entire history (~192 KiB of KV per token β€” 6+ GiB read per generated token at 61K). The window caps each layer's KV, so decode cost flattens instead of growing, and the f16 KV path (measured 23% faster than q8_0 on this kernel despite 2Γ— the bytes) becomes affordable at 61K. We also measured and rejected the "obvious" tricks: KV quantization below f16 is slower here (dequant cost > bytes saved), and bigger micro-batches only help prefill.

Files in this repo

file what
k2-fast2-swa.patch the fork patch (51 lines, applies cleanly to stock MBZUAI model/K2Horizon)
make_swa_gguf.py bake the window key into any k2-horizon GGUF (choose your own W)
fast2-serve.sh tuned launcher (VRAM-gated, ctx-aware, FAST2_SWA support)
results-head2head.json every measurement behind the claims
checksums.json sha256 of the benchmarked artifact + these tools

Honesty clause

Every claim comes from gated, interleaved, repeated measurement published in results-head2head.json β€” the same discipline that showed the previous "FAST" build of this model was not faster at all. FAST2 changes no weights (tensors byte-identical to APEX-Mini, sha-verified), does not touch short-context speed, and removes attention recall beyond the window by design. Provenance: base weights IFM K2-Horizon-MoVA-36B-A4B (apache-2.0); APEX-Mini quant by the reference repo; measured on RTX 2080 Ti 22 GB.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for VladHong/K2-Horizon-MoVA-36B-A4B-FAST2

Finetuned
(15)
this model