HGRN-1.3B · AWP 30% sparse
HGRN-1.3B, a 1.3B gated linear-recurrent LM with no attention, pruned one-shot to 30% weight sparsity with AWP. It is a baseline in the CWS comparison on this model.
Model details
| Base model | ikimyaii/hgrn-1.3B-dense-baseline |
| Method | AWP |
| Target / achieved sparsity | 30% / 30.02% |
| Calibration | dataset not recorded, 64 batches × 4 × 512 tokens |
| Weight correction | none (mask search only) |
| Retraining | none |
What is pruned: every nn.Linear in the 24 HGRN blocks (i_proj, f_proj, g_proj, o_proj, gate_proj, up_proj, down_proj) and lm_head. The same fraction is removed from every output row (semi-structured, constant k per row). Embeddings and norms stay dense.
Method
AWP (Liu et al., 2025, arXiv:2506.10205) minimises the activation-weighted reconstruction error ‖(W − Θ) C^{1/2}‖²_F, C = E[xxᵀ], by iterative hard thresholding: Z = Θ + η (W − Θ) C, η = 2/‖C‖_F, then keep the top-k entries per row. It is initialised from the Wanda mask and runs up to 200 iterations (tolerance 1e-4).
Evaluation
One-shot WikiText-2 perplexity at 30% sparsity on HGRN-1.3B (dense 11.8), as reported in the CWS paper sweep (results/paper_reported_sweep.csv):
| Method | WikiText-2 PPL ↓ |
|---|---|
| CWS | 12.14 |
| SparseGPT | 15.02 |
| Wanda | 31.64 |
| RIA | 25.6 |
| AWP (this model) | 21.7 |
Usage
The model uses the HGRN architecture from flash-linear-attention (FLA).
pip install flash-linear-attention transformers torch
import torch
import fla # registers HGRN with transformers
from fla.models.hgrn import HGRNConfig, HGRNForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
AutoConfig.register("hgrn", HGRNConfig, exist_ok=True)
AutoModelForCausalLM.register(HGRNConfig, HGRNForCausalLM, exist_ok=True)
model_id = "ikimyaii/hgrn-1.3B-awp-30pct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()
# Pruned weights are stored as explicit zeros in a dense tensor.
lin = model.model.layers[0].mlp.down_proj
print(f"sparsity: {(lin.weight == 0).float().mean():.3f}")
The checkpoint is stored densely, with zeros in the pruned positions, so it runs as-is on GPUs and CPUs. Speedups require a sparse kernel or hardware that skips zero weights. On Intel Loihi and other event-driven neuromorphic hardware, a zero weight generates no synaptic event, so the sparsity directly reduces active computation.
Limitations
- These are research checkpoints for studying pruning. They are not instruction-tuned or safety-tuned, and they inherit the base model's biases.
- Weights are stored densely, with zeros at the pruned positions. Sparsity alone gives no speedup without a sparse kernel or event-driven hardware.
Related repositories
- Sparse-LLMs-on-Loihi: pruning scripts, baselines and raw results for this checkpoint.
- CWS: reference implementation of the CWS method these baselines are compared against.
- Weight-Sparsity-Using-Real-Loss and EHWS: follow-up work that optimises the real next-token loss instead of layer-wise reconstruction.
- Downloads last month
- 13
Model tree for ikimyaii/hgrn-1.3B-awp-30pct
Base model
ikimyaii/hgrn-1.3B-dense-baseline