HGRN-1.3B · AWP 30% sparse

HGRN-1.3B, a 1.3B gated linear-recurrent LM with no attention, pruned one-shot to 30% weight sparsity with AWP. It is a baseline in the CWS comparison on this model.

Model details

Base model ikimyaii/hgrn-1.3B-dense-baseline
Method AWP
Target / achieved sparsity 30% / 30.02%
Calibration dataset not recorded, 64 batches × 4 × 512 tokens
Weight correction none (mask search only)
Retraining none

What is pruned: every nn.Linear in the 24 HGRN blocks (i_proj, f_proj, g_proj, o_proj, gate_proj, up_proj, down_proj) and lm_head. The same fraction is removed from every output row (semi-structured, constant k per row). Embeddings and norms stay dense.

Method

AWP (Liu et al., 2025, arXiv:2506.10205) minimises the activation-weighted reconstruction error ‖(W − Θ) C^{1/2}‖²_F, C = E[xxᵀ], by iterative hard thresholding: Z = Θ + η (W − Θ) C, η = 2/‖C‖_F, then keep the top-k entries per row. It is initialised from the Wanda mask and runs up to 200 iterations (tolerance 1e-4).

Evaluation

One-shot WikiText-2 perplexity at 30% sparsity on HGRN-1.3B (dense 11.8), as reported in the CWS paper sweep (results/paper_reported_sweep.csv):

Method WikiText-2 PPL ↓
CWS 12.14
SparseGPT 15.02
Wanda 31.64
RIA 25.6
AWP (this model) 21.7

Usage

The model uses the HGRN architecture from flash-linear-attention (FLA).

pip install flash-linear-attention transformers torch
import torch
import fla  # registers HGRN with transformers
from fla.models.hgrn import HGRNConfig, HGRNForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

AutoConfig.register("hgrn", HGRNConfig, exist_ok=True)
AutoModelForCausalLM.register(HGRNConfig, HGRNForCausalLM, exist_ok=True)

model_id = "ikimyaii/hgrn-1.3B-awp-30pct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()

# Pruned weights are stored as explicit zeros in a dense tensor.
lin = model.model.layers[0].mlp.down_proj
print(f"sparsity: {(lin.weight == 0).float().mean():.3f}")

The checkpoint is stored densely, with zeros in the pruned positions, so it runs as-is on GPUs and CPUs. Speedups require a sparse kernel or hardware that skips zero weights. On Intel Loihi and other event-driven neuromorphic hardware, a zero weight generates no synaptic event, so the sparsity directly reduces active computation.

Limitations

  • These are research checkpoints for studying pruning. They are not instruction-tuned or safety-tuned, and they inherit the base model's biases.
  • Weights are stored densely, with zeros at the pruned positions. Sparsity alone gives no speedup without a sparse kernel or event-driven hardware.

Related repositories

  • Sparse-LLMs-on-Loihi: pruning scripts, baselines and raw results for this checkpoint.
  • CWS: reference implementation of the CWS method these baselines are compared against.
  • Weight-Sparsity-Using-Real-Loss and EHWS: follow-up work that optimises the real next-token loss instead of layer-wise reconstruction.
Downloads last month
13
Safetensors
Model size
1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ikimyaii/hgrn-1.3B-awp-30pct

Finetuned
(22)
this model

Datasets used to train ikimyaii/hgrn-1.3B-awp-30pct

Paper for ikimyaii/hgrn-1.3B-awp-30pct