shisa-ai/Nanbeige4.3-33B-DMS-2x

Nanbeige4.3-33B DMS sidecar — CR2, W8192

A standalone 5.7 MiB eviction-head sidecar, not a language-model checkpoint. It adds learned per-KV-head token eviction to Nanbeige without changing its base weights or borrowing query channels. This release contains the selected 64K, epoch-10 candidate, trained from real Shisa-v2.1 and Sansho data using the exact primitive-ai FP8 serving model.

• Engine and training code: shisa-ai/FastDMS.
• Original model: Nanbeige/Nanbeige4.3-33B.
• Evaluated FP8 checkpoint: primitive-ai/Nanbeige4.3-33B-FP8.

Measured result: all four 32K/128K synthetic retrieval cases recovered every target token under teacher forcing with both BF16 and FP8 KV. On six previously sealed real-data 64K contexts, prediction agreement with dense BF16 KV was 94.09% for DMS BF16 KV and 91.94% for DMS FP8 KV. The full results below include regressions and the fresh-data continuation; this is not a claim of universal quality equivalence.

Load with FastDMS

Install FastDMS following its installation instructions. A compatible FlashAttention build is required for serving. Download the sidecar separately and pass its metadata path:

from huggingface_hub import snapshot_download
from fastdms import LLM, SamplingParams

sidecar = snapshot_download("shisa-ai/Nanbeige4.3-33B-DMS-2x")
llm = LLM(
    "primitive-ai/Nanbeige4.3-33B-FP8",
    enforce_eager=True,
    max_model_len=32768,
    max_num_batched_tokens=32768,
    max_num_seqs=1,
    gpu_memory_utilization=0.90,
    compact_kv_enabled=True,
    compact_kv_capacity_tokens=8 * 32768,
    compact_kv_retention_mode="dms",
    dms_metadata_path=f"{sidecar}/dms_metadata.json",
    kv_cache_dtype="bfloat16",  # or "float8_e4m3fn"
)
outputs = llm.generate(
    ["Explain Rayleigh scattering."],
    SamplingParams(temperature=0, max_tokens=128),
)
print(outputs[0]["text"])

Capacity is head-token slots per layer, shared across KV heads; it is not a per-head token limit. Size it for the intended total request workload. This example allocates capacity for one request. max_model_len includes prompt and generated tokens. The tests used one RTX PRO 6000 Blackwell 96 GB GPU per serving process, PyTorch 2.11.0+cu130, and Transformers 5.12.1.

Format and hipEngine compatibility

The package uses hipEngine's schema-v2 external_linear_sidecar_v1 contract:

• BF16 safetensors weight: [52, 8, 7168]; bias: [52, 8].
• 2,982,304 parameters; normalized input stage post_attn_rmsnorm_pre_q_projection.
• Physical layers 0–51; 56 query heads, 8 KV heads, head dimension 128.
• Ordinary query channels remain unchanged.
• Prefill uses exact_budget: retain half of history outside the protected window independently per head. Decode uses calibrated threshold decisions.
• Window 8192: keys with age at most 8192 remain protected. Total live compression is below 2x because the protected window is not compressed.

The selected export was loaded by the actual hipEngine schema-v2 loader. Tests compare projections and future-attention labels against hipEngine references. This establishes format/CPU-contract compatibility, not AMD serving qualification for Nanbeige. FastDMS supports both external sidecars and its existing baked-in borrowed-query DMS checkpoints.

The artifact fingerprint binds a manifest of the FP8 checkpoint's config and weight-shard hashes, not a GGUF hash. Applying it to a different checkpoint, quantization or revision is not covered by these measurements. FastDMS validates sidecar SHA-256, geometry and semantics; it does not automatically rehash the entire base checkpoint on every load. Use the revisions in provenance.json.

Training data and procedure

Six sources: Shisa-v2.1 SFT conversations plus Sansho's cached FineWiki English, FineWiki Japanese, Stack-Edu Python, PG19 and Swallow Math v2 textbooks. Nanbeige's tokenizer was used throughout. Training text, token streams and captured hidden states are not distributed in this repository.

The 64K corpus has 48 training streams (3,145,728 tokens), six validation streams and six final streams. Text SHA-256 assigns document splits; exact duplicates are removed. This is exact-document separation, not near-duplicate or semantic decontamination. Final streams were opened only after the baseline and continuation checkpoints were frozen.

Capture runs the exact FP8 model with no base-weight updates. Labels rank future attention mass per KV head outside the protected window. Full contexts are processed, with query stride16 and 1,024 sampled eligible hidden rows per layer per stream. This is a query-subsampled oracle, not full stride-one attention labeling. Each epoch contains 20,447,232 training label decisions.

Training uses independent linear heads, AdamW, LR 0.001, batch512, zero weight decay, binary cross-entropy plus a 0.1 budget penalty, and gradient norm cap1. Calibration uses training rows only, after rounding weights to BF16. The selected checkpoint is epoch10 of the 64K run. A subsequent ten-epoch run used 24 fresh and 24 replay streams per epoch, adding 15,728,640 fresh tokens. It is reported below but is not the released checkpoint.

See RESULTS.md for training, quality, speed and memory tables and PERFORMANCE.md for the full matched speed table of all four real-data candidates. Raw scalar results are in results/. provenance.json records checkpoint revisions, hashes, source-code identity and the selection decision.

Evaluation protocol and limitations

Quality uses identical teacher-forced continuations and dense BF16 KV as the reference, holding the primitive-ai FP8 weights fixed. The first prediction is excluded from decode agreement/KL because all variants use dense prefill. Synthetic retrieval checks cover 32K/128K with an early/late six-digit code. Real validation and final sets each contain six 64K contexts and 31 scored decode predictions per context. These are bounded tests, not free-generation benchmark scores.

Performance uses batch1, eager execution, a repeated explanatory-text prompt, 32 output tokens for long-context runs, one warmup and two timed repeats. Prefix cache is cleared. Memory is PyTorch allocated peak during inference, not total process memory or model-load peak. Clocks were not locked; some runs shared the host with workloads on other GPUs. Small speed differences are recorded measurements, not statistically established improvements.

Training contexts were 32K and 64K. Serving was tested through 128K; no 128K training run was performed. No claim is made about multi-request throughput, tensor-parallel end-to-end qualification, or cross-quantization quality.

License

The sidecar weights are released under Apache-2.0. FastDMS code is MIT-licensed. The base models and training sources retain their own licenses. No base-model weights or training corpus are included.