Qwen3.8-27B Q4_K_M DMS W8192
A 655,640-byte BF16 Dynamic Memory Sparsification (DMS)
sidecar trained against the pinned
unsloth/Qwen3.8-27B-GGUF
Q4_K_M quant for hipEngine. The
repository contains no target-model weights.
Runtime metadata currently binds this sidecar to an exact GGUF SHA; transfer to another quantization of the same checkpoint is plausible, but currently remains unqualified.
At 128K, the explicit c1 route reduces full-attention BF16 K/V payload
8.000 -> 4.250 GiB (1.882225x) at 136.27 ms/token versus 137.43 ms/token for
dense BF16 K/V. shisa-ai/FastDMS
provides the CUDA compact-DMS implementation and serving lineage.
Measured 2026-08-24 on AMD Ryzen AI Max+ PRO 395 / Radeon 8060S, ROCm 7.15.0:
| metric | DMS | dense BF16-KV control |
|---|---|---|
| full-attention KV payload | 4.250 GiB | 8.000 GiB |
| payload reduction | 3.750 GiB / 1.882225x | 1.000x |
| median decode latency | 136.27 ms/token | 137.43 ms/token |
| reciprocal decode rate | 7.34 tok/s | 7.28 tok/s |
| strict-teacher max / mean KL | 0.002899 / 0.000321 | reference |
| strict-teacher top-1 agreement | 32/32 (100%) | reference |
This is a c=1 token-latency measurement, excluding HTTP serving and concurrency.
Artifact binding
| field | required value |
|---|---|
| base model | Qwen/Qwen3.8-27B |
| GGUF | unsloth/Qwen3.8-27B-GGUF @ 4121cb19, Qwen3.8-27B-Q4_K_M.gguf |
| GGUF bytes | 17,106,775,008 |
| GGUF SHA-256 | 7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169 |
| model family | qwen35_dense_hybrid |
| hidden size | 5120 |
| full-attention layers | physical layers 3,7,11,...,63 (16 total) |
| Q / KV heads | 24 / 4 |
| head dimension | 256 |
| qualified contexts | 32768, 131072 text tokens |
| engine | shisa-ai/hipEngine commit 6d291bf16 or descendant |
| backend | hip_gfx1151 |
dms_metadata.json binds the complete GGUF SHA-256, sidecar SHA-256, tensor
names/shapes, physical layer map, predictor input stage, and policy. Hash,
geometry, or provenance mismatches fail before sidecar execution.
KV eviction path
For compact full-attention layer l, KV head h, and normalized pre-Q hidden
row x:
score[l,h] = dot(weight[l,h], x) + bias[l,h]
evict[l,h] = score[l,h] * 1.0 - 0.012540675699710846 > 0
Predictor tensors:
weight: [16, 4, 5120] BF16
bias: [16, 4] BF16
The sidecar consumes post_attn_rmsnorm_pre_q_projection; ordinary Q channels
are unchanged. Prefill uses learned scores as deterministic per-layer/head
ranks, applies an exact historical CR2 budget, and protects the newest 8,192
tokens. Decode evaluates the sidecar on device, applies window expiry, and
appends surviving K/V directly to per-head compact extents.
The attention ABI carries base_offsets, live_counts, token_positions, and
evict_mask. The Qwen3.8 production kernel assigns one compact token to each
wave, distributes its 256-dimensional K dot product over 32 lanes, and shares K
loads across the six query heads mapped to one KV head. At 69,641 live rows per
KV head, the producer median is 1.422 ms/layer, 9.792x below the previous
grouped producer. The kernel uses 1,092 local256 workgroups, 56 VGPR,
12,800-byte LDS, and zero scratch; a 0.095 ms split reducer follows it.
Prefill writes temporary dense BF16 K/V, packs the selected rows, then releases the dense owner. Decode holds no dense K/V mirror. The temporary prefill owner remains part of peak memory.
DMS lineage
| source | role |
|---|---|
| DMS paper: Inference-Time Hyper-Scaling with KV Cache Compression | learned per-head eviction objective and retention-window policy |
NVIDIA Model-Optimizer experimental/dms |
public DMS retrofit/training reference |
shisa-ai/FastDMS |
CUDA compact-KV serving implementation, expiry semantics, metadata contract, and split-K attention reference |
shisa-ai/hipEngine |
external sidecar training, exact-budget W8192 policy, KVLiveSpans ownership, and ROCm kernels used for this package |
FastDMS checkpoint retrofits store eviction signals in a borrowed query channel.
This package uses an external linear sidecar; borrowed_query_channel is null
and zero_borrowed_query_channel is false. hipEngine development referenced
FastDMS commit c602b0ec3266da7f74d6a658b3dafcddb443fddd read-only.
Quality and capacity
Protocol:
- exact Q4_K_M target above;
- dense BF16-KV exact-Q4 resident teacher;
- teacher token trajectory and full-vocabulary logits;
- categories: code, English, Japanese, mixed Japanese/English;
- eight decode steps per category;
- acceptance: max KL <= 0.05 and top-1 agreement >= 90% in every category;
- 32K sealed validation plus a fresh source-disjoint 128K validation corpus.
| context | rows | max / mean KL | p95 / p99 KL | top-1 | live rows | max live/head | live CR | compact BF16 K/V |
|---|---|---|---|---|---|---|---|---|
| 32K | 32 | 0.003784 / 0.000333 | 0.000706 / 0.002856 | 100% | 1,311,296 | 20,489 | 1.599688x | 1,342,767,104 B |
| 128K | 32 | 0.002899 / 0.000321 | 0.001067 / 0.002419 | 100% | 4,457,024 | 69,641 | 1.882225x | 4,563,992,576 B |
All four categories pass independently. The 128K corpus has zero source overlap with the training, 32K development, and 32K sealed-final corpora.
Observed CR is below 2x because the 8,192-token window remains live. The payload rows cover the 16 full-attention layers; recurrent Gated DeltaNet state and runtime scratch are separate.
Decode timing
| context | wave DMS | prior grouped DMS | dense BF16-KV | wave / dense |
|---|---|---|---|---|
| 32K | 104.62 ms/token | 192.74 ms/token | 99.96 ms/token | 1.0467x |
| 128K | 136.27 ms/token | 357.96 ms/token | 137.43 ms/token | 0.9916x |
The wave kernel improves the retained grouped route 1.842x at 32K and 2.627x at 128K. The 32K row remains 4.67% above dense latency. No automatic context threshold is packaged.
Training
DMS sidecars are trained per model checkpoint. These weights were
optimized on one Strix Halo system (AMD Ryzen AI Max+ PRO 395 / Radeon 8060S,
gfx1151) for 682.491 seconds (11m22s) using PyTorch
2.10.0+rocm7.15.0a20260727. This is optimizer time: Q/K capture, exact
future-attention labeling, rejected calibration experiments, and quality gates
are separate.
Only weight and bias were optimized. The GGUF bytes were unchanged.
| item | value |
|---|---|
| long calibration sequences | 4 x 32K |
| labeled sequence tokens | 131,072 |
| label query stride | 1 over full causal history |
| full-attention label shards | 64 |
| long fine-tune epochs | 20 |
| short initialization optimizer time | 99.418 s |
| exact 32K fine-tune optimizer time | 583.073 s |
| retained-weight optimizer total | 682.491 s (11m22s) |
| long fine-tune peak device allocation | 793,179,136 B (756.43 MiB) |
| sidecar parameters | 327,744 |
| sidecar bytes | 655,640 |
A train-only per-head threshold calibration ran for 637.078 s and produced
sidecar SHA d2fe336e...6f23d8b. That policy failed the 32K development gate and
was discarded. This package contains epoch-20 sidecar e52fc60a...cd764.
W8192/exact-budget changed metadata and selection policy without another
optimizer run.
Training depth and decode speed
At fixed CR2/W8192 exact_budget, additional epochs do not reduce the survivor
count and therefore do not reduce compact-attention work. They can improve which
tokens survive and increase quality margin. A speed gain requires qualifying a
higher compression ratio, a smaller protected window, or a different policy
that retains fewer rows.
The long fine-tune reached 83.60% internal decision accuracy and 0.3660 BCE over 520,192 reserved decisions. Its threshold-calibrated CR2 policy still failed the Japanese 32K development category at max KL 0.14007 and 87.5% top-1. The short checkpoint's CR4/CR8 modes were also rejected at max KL 0.08908/0.24993. W8192/exact-budget is the retained quality/speed point; additional training has no retained speedup evidence beyond it.
Training/calibration text sources:
| source | license |
|---|---|
| CPython 3.13 standard-library source | PSF-2.0 |
| Wikimedia English/Japanese Wikipedia snapshots | CC-BY-SA-3.0 and GFDL |
Benchmark prompts were excluded from training. The package contains no source text, captures, labels, GGUF weights, or evaluation prompts.
Files
| file | bytes | SHA-256 |
|---|---|---|
qwen38-27b-q4km-dms-sidecar.safetensors |
655,640 | e52fc60a4e8f86f00719be107b44a73567795dfdd88dbbce15cf955409fcd764 |
dms_metadata.json |
2,090 | 62c1f2d2c590aff2756796f01bddfaf1b602750ceada4b3ac23639902ff8edfe |
qualification.json |
see manifest | portable protocol/results record |
MANIFEST.json |
see file | SHA-256 and byte count for every package file except itself |
hf download shisa-ai/Qwen3.8-27B-Q4_K_M-DMS-W8192 \
--local-dir ./Qwen3.8-27B-Q4_K_M-DMS-W8192
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-Q4_K_M.gguf \
--revision 4121cb19390a7984a0e6dc0f46bea9177b846f15 \
--local-dir ./Qwen3.8-27B-Q4_K_M
sha256sum \
./Qwen3.8-27B-Q4_K_M-DMS-W8192/qwen38-27b-q4km-dms-sidecar.safetensors \
./Qwen3.8-27B-Q4_K_M-DMS-W8192/dms_metadata.json \
/path/to/Qwen3.8-27B-Q4_K_M.gguf
hipEngine route
export HIPENGINE_HIP_ARCH=gfx1151
from pathlib import Path
from hipengine.runtime.qwen35_gguf_runner import Qwen35GGUFResidentSession
model = Path("/path/to/Qwen3.8-27B-Q4_K_M.gguf")
package = Path("./Qwen3.8-27B-Q4_K_M-DMS-W8192")
max_new_tokens = 128
with Qwen35GGUFResidentSession(
model,
backend="hip_gfx1151",
max_sequence_length=131072 + max_new_tokens,
dms_metadata_path=package / "dms_metadata.json",
dms_max_new_tokens=max_new_tokens + 1,
use_wmma_prefill=True,
use_gemv_decode=True,
) as session:
result = session.prefill(
prompt_token_ids,
use_bulk=True,
bulk_attention_mode="bulk",
)
prompt_token_ids must fit max_sequence_length. The qualified harness is
hipEngine/scripts/qwen38_dms_integrated_quality_suite.py. Session construction
verifies the GGUF and sidecar fingerprints before allocation.
Compatibility matrix
| target | status |
|---|---|
| exact Q4_K_M GGUF SHA above, gfx1151, explicit c1 | qualified at 32K/128K |
| Q4_K_S or another GGUF quant/file | unqualified; metadata hash rejects it |
| PARO | no runtime integration or qualification |
| gfx1100 | generic HIP source exists; this production arithmetic is gfx1151-only |
| CUDA / FastDMS runtime | incompatible sidecar convention |
| c2-c32 model serving | allocator fixtures pass; full-model lifecycle/soak open |
| prefix cache | disabled pending per-sequence eviction overlays |
| MTP / DFlash | disabled pending speculative-span commit/rollback gates |
| vision | untested; hipEngine GGUF path is text-only |
hipengine.LLM.generate() |
package admission open |
Open serving gates
- Full-model c1-c32 admission, cancellation, pressure, recovery, and drain.
- Long c1/c8 soak with sampled process/device memory.
- Streaming compact prefill without the temporary dense K/V peak.
- Public model-plugin package admission and strict dense fallback.
- Same-host c8/c32 performance and the 32K-to-128K break-even context.
License and references
The sidecar is a modified derivative of Apache-2.0-licensed
Qwen/Qwen3.8-27B model behavior and
is distributed under Apache License 2.0. See LICENSE and NOTICE.
hipEngine is AGPL-3.0-or-later. FastDMS is MIT. Neither codebase is included in this package.
Model tree for shisa-ai/Qwen3.8-27B-Q4_K_M-DMS-W8192
Base model
Qwen/Qwen3.8-27B