Qwen3.8-27B Q4_K_M DMS W8192

A 655,640-byte BF16 Dynamic Memory Sparsification (DMS) sidecar trained against the pinned unsloth/Qwen3.8-27B-GGUF Q4_K_M quant for hipEngine. The repository contains no target-model weights.

Runtime metadata currently binds this sidecar to an exact GGUF SHA; transfer to another quantization of the same checkpoint is plausible, but currently remains unqualified.

At 128K, the explicit c1 route reduces full-attention BF16 K/V payload 8.000 -> 4.250 GiB (1.882225x) at 136.27 ms/token versus 137.43 ms/token for dense BF16 K/V. shisa-ai/FastDMS provides the CUDA compact-DMS implementation and serving lineage.

Measured 2026-08-24 on AMD Ryzen AI Max+ PRO 395 / Radeon 8060S, ROCm 7.15.0:

metric DMS dense BF16-KV control
full-attention KV payload 4.250 GiB 8.000 GiB
payload reduction 3.750 GiB / 1.882225x 1.000x
median decode latency 136.27 ms/token 137.43 ms/token
reciprocal decode rate 7.34 tok/s 7.28 tok/s
strict-teacher max / mean KL 0.002899 / 0.000321 reference
strict-teacher top-1 agreement 32/32 (100%) reference

This is a c=1 token-latency measurement, excluding HTTP serving and concurrency.

Artifact binding

field required value
base model Qwen/Qwen3.8-27B
GGUF unsloth/Qwen3.8-27B-GGUF @ 4121cb19, Qwen3.8-27B-Q4_K_M.gguf
GGUF bytes 17,106,775,008
GGUF SHA-256 7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169
model family qwen35_dense_hybrid
hidden size 5120
full-attention layers physical layers 3,7,11,...,63 (16 total)
Q / KV heads 24 / 4
head dimension 256
qualified contexts 32768, 131072 text tokens
engine shisa-ai/hipEngine commit 6d291bf16 or descendant
backend hip_gfx1151

dms_metadata.json binds the complete GGUF SHA-256, sidecar SHA-256, tensor names/shapes, physical layer map, predictor input stage, and policy. Hash, geometry, or provenance mismatches fail before sidecar execution.

KV eviction path

For compact full-attention layer l, KV head h, and normalized pre-Q hidden row x:

score[l,h] = dot(weight[l,h], x) + bias[l,h]
evict[l,h] = score[l,h] * 1.0 - 0.012540675699710846 > 0

Predictor tensors:

weight: [16, 4, 5120] BF16
bias:   [16, 4]       BF16

The sidecar consumes post_attn_rmsnorm_pre_q_projection; ordinary Q channels are unchanged. Prefill uses learned scores as deterministic per-layer/head ranks, applies an exact historical CR2 budget, and protects the newest 8,192 tokens. Decode evaluates the sidecar on device, applies window expiry, and appends surviving K/V directly to per-head compact extents.

The attention ABI carries base_offsets, live_counts, token_positions, and evict_mask. The Qwen3.8 production kernel assigns one compact token to each wave, distributes its 256-dimensional K dot product over 32 lanes, and shares K loads across the six query heads mapped to one KV head. At 69,641 live rows per KV head, the producer median is 1.422 ms/layer, 9.792x below the previous grouped producer. The kernel uses 1,092 local256 workgroups, 56 VGPR, 12,800-byte LDS, and zero scratch; a 0.095 ms split reducer follows it.

Prefill writes temporary dense BF16 K/V, packs the selected rows, then releases the dense owner. Decode holds no dense K/V mirror. The temporary prefill owner remains part of peak memory.

DMS lineage

source role
DMS paper: Inference-Time Hyper-Scaling with KV Cache Compression learned per-head eviction objective and retention-window policy
NVIDIA Model-Optimizer experimental/dms public DMS retrofit/training reference
shisa-ai/FastDMS CUDA compact-KV serving implementation, expiry semantics, metadata contract, and split-K attention reference
shisa-ai/hipEngine external sidecar training, exact-budget W8192 policy, KVLiveSpans ownership, and ROCm kernels used for this package

FastDMS checkpoint retrofits store eviction signals in a borrowed query channel. This package uses an external linear sidecar; borrowed_query_channel is null and zero_borrowed_query_channel is false. hipEngine development referenced FastDMS commit c602b0ec3266da7f74d6a658b3dafcddb443fddd read-only.

Quality and capacity

Protocol:

  • exact Q4_K_M target above;
  • dense BF16-KV exact-Q4 resident teacher;
  • teacher token trajectory and full-vocabulary logits;
  • categories: code, English, Japanese, mixed Japanese/English;
  • eight decode steps per category;
  • acceptance: max KL <= 0.05 and top-1 agreement >= 90% in every category;
  • 32K sealed validation plus a fresh source-disjoint 128K validation corpus.
context rows max / mean KL p95 / p99 KL top-1 live rows max live/head live CR compact BF16 K/V
32K 32 0.003784 / 0.000333 0.000706 / 0.002856 100% 1,311,296 20,489 1.599688x 1,342,767,104 B
128K 32 0.002899 / 0.000321 0.001067 / 0.002419 100% 4,457,024 69,641 1.882225x 4,563,992,576 B

All four categories pass independently. The 128K corpus has zero source overlap with the training, 32K development, and 32K sealed-final corpora.

Observed CR is below 2x because the 8,192-token window remains live. The payload rows cover the 16 full-attention layers; recurrent Gated DeltaNet state and runtime scratch are separate.

Decode timing

context wave DMS prior grouped DMS dense BF16-KV wave / dense
32K 104.62 ms/token 192.74 ms/token 99.96 ms/token 1.0467x
128K 136.27 ms/token 357.96 ms/token 137.43 ms/token 0.9916x

The wave kernel improves the retained grouped route 1.842x at 32K and 2.627x at 128K. The 32K row remains 4.67% above dense latency. No automatic context threshold is packaged.

Training

DMS sidecars are trained per model checkpoint. These weights were optimized on one Strix Halo system (AMD Ryzen AI Max+ PRO 395 / Radeon 8060S, gfx1151) for 682.491 seconds (11m22s) using PyTorch 2.10.0+rocm7.15.0a20260727. This is optimizer time: Q/K capture, exact future-attention labeling, rejected calibration experiments, and quality gates are separate.

Only weight and bias were optimized. The GGUF bytes were unchanged.

item value
long calibration sequences 4 x 32K
labeled sequence tokens 131,072
label query stride 1 over full causal history
full-attention label shards 64
long fine-tune epochs 20
short initialization optimizer time 99.418 s
exact 32K fine-tune optimizer time 583.073 s
retained-weight optimizer total 682.491 s (11m22s)
long fine-tune peak device allocation 793,179,136 B (756.43 MiB)
sidecar parameters 327,744
sidecar bytes 655,640

A train-only per-head threshold calibration ran for 637.078 s and produced sidecar SHA d2fe336e...6f23d8b. That policy failed the 32K development gate and was discarded. This package contains epoch-20 sidecar e52fc60a...cd764. W8192/exact-budget changed metadata and selection policy without another optimizer run.

Training depth and decode speed

At fixed CR2/W8192 exact_budget, additional epochs do not reduce the survivor count and therefore do not reduce compact-attention work. They can improve which tokens survive and increase quality margin. A speed gain requires qualifying a higher compression ratio, a smaller protected window, or a different policy that retains fewer rows.

The long fine-tune reached 83.60% internal decision accuracy and 0.3660 BCE over 520,192 reserved decisions. Its threshold-calibrated CR2 policy still failed the Japanese 32K development category at max KL 0.14007 and 87.5% top-1. The short checkpoint's CR4/CR8 modes were also rejected at max KL 0.08908/0.24993. W8192/exact-budget is the retained quality/speed point; additional training has no retained speedup evidence beyond it.

Training/calibration text sources:

source license
CPython 3.13 standard-library source PSF-2.0
Wikimedia English/Japanese Wikipedia snapshots CC-BY-SA-3.0 and GFDL

Benchmark prompts were excluded from training. The package contains no source text, captures, labels, GGUF weights, or evaluation prompts.

Files

file bytes SHA-256
qwen38-27b-q4km-dms-sidecar.safetensors 655,640 e52fc60a4e8f86f00719be107b44a73567795dfdd88dbbce15cf955409fcd764
dms_metadata.json 2,090 62c1f2d2c590aff2756796f01bddfaf1b602750ceada4b3ac23639902ff8edfe
qualification.json see manifest portable protocol/results record
MANIFEST.json see file SHA-256 and byte count for every package file except itself
hf download shisa-ai/Qwen3.8-27B-Q4_K_M-DMS-W8192 \
  --local-dir ./Qwen3.8-27B-Q4_K_M-DMS-W8192

hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-Q4_K_M.gguf \
  --revision 4121cb19390a7984a0e6dc0f46bea9177b846f15 \
  --local-dir ./Qwen3.8-27B-Q4_K_M

sha256sum \
  ./Qwen3.8-27B-Q4_K_M-DMS-W8192/qwen38-27b-q4km-dms-sidecar.safetensors \
  ./Qwen3.8-27B-Q4_K_M-DMS-W8192/dms_metadata.json \
  /path/to/Qwen3.8-27B-Q4_K_M.gguf

hipEngine route

export HIPENGINE_HIP_ARCH=gfx1151
from pathlib import Path
from hipengine.runtime.qwen35_gguf_runner import Qwen35GGUFResidentSession

model = Path("/path/to/Qwen3.8-27B-Q4_K_M.gguf")
package = Path("./Qwen3.8-27B-Q4_K_M-DMS-W8192")
max_new_tokens = 128

with Qwen35GGUFResidentSession(
    model,
    backend="hip_gfx1151",
    max_sequence_length=131072 + max_new_tokens,
    dms_metadata_path=package / "dms_metadata.json",
    dms_max_new_tokens=max_new_tokens + 1,
    use_wmma_prefill=True,
    use_gemv_decode=True,
) as session:
    result = session.prefill(
        prompt_token_ids,
        use_bulk=True,
        bulk_attention_mode="bulk",
    )

prompt_token_ids must fit max_sequence_length. The qualified harness is hipEngine/scripts/qwen38_dms_integrated_quality_suite.py. Session construction verifies the GGUF and sidecar fingerprints before allocation.

Compatibility matrix

target status
exact Q4_K_M GGUF SHA above, gfx1151, explicit c1 qualified at 32K/128K
Q4_K_S or another GGUF quant/file unqualified; metadata hash rejects it
PARO no runtime integration or qualification
gfx1100 generic HIP source exists; this production arithmetic is gfx1151-only
CUDA / FastDMS runtime incompatible sidecar convention
c2-c32 model serving allocator fixtures pass; full-model lifecycle/soak open
prefix cache disabled pending per-sequence eviction overlays
MTP / DFlash disabled pending speculative-span commit/rollback gates
vision untested; hipEngine GGUF path is text-only
hipengine.LLM.generate() package admission open

Open serving gates

  1. Full-model c1-c32 admission, cancellation, pressure, recovery, and drain.
  2. Long c1/c8 soak with sampled process/device memory.
  3. Streaming compact prefill without the temporary dense K/V peak.
  4. Public model-plugin package admission and strict dense fallback.
  5. Same-host c8/c32 performance and the 32K-to-128K break-even context.

License and references

The sidecar is a modified derivative of Apache-2.0-licensed Qwen/Qwen3.8-27B model behavior and is distributed under Apache License 2.0. See LICENSE and NOTICE.

hipEngine is AGPL-3.0-or-later. FastDMS is MIT. Neither codebase is included in this package.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for shisa-ai/Qwen3.8-27B-Q4_K_M-DMS-W8192

Base model

Qwen/Qwen3.8-27B
Finetuned
(983)
this model

Paper for shisa-ai/Qwen3.8-27B-Q4_K_M-DMS-W8192