DMS Qwen3-1.7B β€” 8x KV-cache compression (adapters only)

Dynamic Memory Sparsification (DMS) adapters trained on Qwen/Qwen3-1.7B with the unmodified upstream recipe from NVIDIA/Model-Optimizer experimental/dms, pinned at commit 73d778422388f0e849ecb180375d34ac445711ca.

What is in this repo

Adapters only β€” the base weights are not duplicated here. DMS freezes the base model and trains only the per-head eviction adapters (dms_proj_alpha, dms_proj_alpha_norm; 56 tensors, ~1 MB). Before publishing, every non-DMS tensor of the trained student was compared against the public base checkpoint and found bit-identical, and the round trip base + adapters -> student was executed and verified tensor for tensor (366/366 identical, 0 differing, 0 missing). Reconstructing from this repo gives you exactly the checkpoint that was trained, not an approximation of it.

Loading

The modeling code imports the dms package, so install it first:

git clone https://github.com/NVIDIA/Model-Optimizer
cd Model-Optimizer/experimental/dms && pip install -e .

Then, from a clone of this repo:

python load_dms.py --out ./dms_student     # pulls Qwen/Qwen3-1.7B, applies the adapters, saves
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("./dms_student", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("./dms_student")

AutoModelForCausalLM.from_pretrained("<this repo>", trust_remote_code=True) will not work directly: there are no base weights here, and trust_remote_code resolves auto_map against the repo being loaded rather than the one the config came from. load_dms.py handles both issues; see its docstring for the detail.

Training

base Qwen/Qwen3-1.7B (frozen)
target compression final_cr = 8.0 (dms_cr = 1/(1-closed_frac))
context / sliding window 8192 / 512
data OpenR1-Math-220k, 4000 samples
steps 544 (CR ramp over the first 510)
effective batch 16
optimizer AdamW, lr 3e-5 constant, beta2 0.95, bf16
hardware / wall clock 1x H200, 11.5 h

The full config is in train_config.yaml. Objective is KL distillation against the frozen base as teacher, plus a compression hinge max(0, target_frac - closed_frac).

Measured at the end of training

metric last step mean of last 32 steps
dms_cr (achieved compression) 8.135 8.361
dms_target_cr 8.0 8.0
distillation loss (KL vs base) 0.07438 0.05466
detached LM loss 1.0857 0.7155

The compression target is met. These adapters have not been benchmarked on downstream tasks β€” no LongBench, RULER, or math-reasoning numbers are claimed here. Treat the KL figure as a training diagnostic, not as evidence of task-level quality.

License

Apache-2.0, matching both Qwen/Qwen3-1.7B and NVIDIA Model-Optimizer.

Downloads last month
15
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for faridlazuarda/dms-qwen3-1.7b-cr8

Finetuned
Qwen/Qwen3-1.7B
Finetuned
(1,428)
this model

Paper for faridlazuarda/dms-qwen3-1.7b-cr8