Seed-OSS-36B-Base GPTQ 4-bit (W4A16)

This is a 4-bit GPTQ quantized version of ByteDance-Seed/Seed-OSS-36B-Base using LLM Compressor, saved in the compressed-tensors format that vLLM loads natively.

Key Features

  • 36B parameters quantized to 4-bit - 71% size reduction (72.3 GB → 21.0 GB)
  • Fits on a single GPU - weights use 19.9 GiB in vLLM, instead of needing 2× 80 GB GPUs in BF16
  • Every decoder Linear layer quantized with GPTQ, including attention
  • Apache 2.0 base model with a 512K-token context window
  • vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
  • +1.22% perplexity vs BF16 on C4 validation; ARC-Challenge 61.4% vs 61.1%, HellaSwag 81.6% vs 82.3%, GSM8K 83.0% vs 86.7%
  • Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes

Model Details

  • Base Model: ByteDance-Seed/Seed-OSS-36B-Base (36.2B parameters, base/pretrained model, not instruction-tuned; this is the version trained with synthetic instruction data, not the -woSyn variant)
  • Architecture: Seed-OSS (SeedOssForCausalLM), 64 layers, hidden size 5120, grouped-query attention (80 query heads, 8 KV heads, head dim 128), MLP size 27,648, vocabulary 155,136, biases on the q/k/v projections
  • Quantization Method: GPTQ
  • Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
  • Group Size: 128, symmetric, static activation ordering, dampening 0.01
  • Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)

Size Comparison

Metric Value
Original (BF16) ~72.3 GB
Quantized (W4A16) ~21.0 GB
Reduction ~71%
Memory Saved ~51.3 GB

The quantized share is smaller than for models with smaller vocabularies: the 155K-token embedding table and lm_head stay in BF16 (3.2 GB of the 21.0 GB).

What Was Quantized

Quantized (4-bit):

  • All 448 decoder Linear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj in all 64 layers

Preserved (BF16):

  • lm_head
  • Token embeddings
  • Normalization layers
  • Attention projection biases

How It Was Quantized: Multi-Node Distributed GPTQ

This model was quantized on Kubernetes (Amazon EKS, us-west-2) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:

Step Detail
Launch Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; torchrun with 8 processes per node (16 ranks)
Interconnect NVLink inside each node; EFA between nodes via aws-ofi-nccl
Storage BF16 weights and calibration data read from an S3 bucket mounted as a Kubernetes PersistentVolume (Mountpoint for Amazon S3 CSI driver); the checkpoint was written to a second S3-backed PersistentVolume. No pod downloaded the model
Model loading Rank 0 reads the BF16 weights from the volume and broadcasts replicas to all 16 ranks over NCCL (240 s)
Calibration Each rank runs forward passes on its own 16-document slice (data-parallel calibration)
Compression Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points
Runtime 179 s for all 65 sequential stages; peak GPU memory 77.1 GiB (highest across ranks)

Notes from the run:

  • Memory. Every rank holds a full BF16 replica (67.3 GiB), which leaves little room for GPTQ on an 80 GB H100. The embedding table and lm_head (3.0 GiB, untouched by GPTQ) were moved to each rank's own CPU memory during calibration and moved back to the GPU for saving. Peak was 77.1 GiB.
  • Multi-node device bug. LLM Compressor 0.14.0's get_main_device() selects the GPU by global rank, which fails on the second node (cuda:8–cuda:15 don't exist on an 8-GPU machine). The run patches it to use the local device index.
  • Offload across nodes. LLM Compressor rebuilds offloading around save_pretrained. Modules left on CPU get a cache shared through /dev/shm, which ranks on another node can't open. Keeping every module on the GPU before saving avoids it.

Evaluation

Tested with vLLM 0.30 on p5.48xlarge nodes (H100 80 GB): this model on 1 GPU, the BF16 base model on 2 GPUs (tensor parallel) as the baseline. vLLM loads this checkpoint with MacheteLinearKernel (compressed-tensors WNA16); weights use 19.85 GiB of GPU memory.

Perplexity

C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):

Model Perplexity
BF16 (original) 9.010
GPTQ W4A16 (this model) 9.120 (+1.22%)

Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16. A Fibonacci function prompt produced the same function body.

Task Accuracy

lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):

Task Metric BF16 GPTQ W4A16 Difference
ARC-Challenge (0-shot) acc_norm 61.09% 61.43% +0.34
HellaSwag (0-shot) acc_norm 82.28% 81.60% −0.68
GSM8K (5-shot) exact_match, flexible-extract 86.66% 83.02% −3.64
GSM8K (5-shot) exact_match, strict 76.65% 63.61% −13.04

Next-Token Agreement

The same 102,300 C4 validation positions, top-20 log-probabilities from each server:

Measure Value
Same top-1 token as BF16 94.39%
BF16's top-1 token within this model's top-5 99.94%
KL(BF16 ‖ GPTQ), mean / median / p99 0.0144 / 0.0073 / 0.129 nats

The KL estimate uses only BF16's top-20 tokens (88.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.

Throughput and Latency

vllm bench serve, random 1,024-token inputs / 256-token outputs, 400 prompts, --max-model-len 8192:

Model GPUs Request rate Output tok/s Median TTFT Median TPOT
GPTQ W4A16 1 1/s 254 164 ms 15.3 ms
GPTQ W4A16 1 4/s 982 271 ms 38.9 ms
GPTQ W4A16 1 16/s 1,230 7,564 ms 119.8 ms
GPTQ W4A16 1 unlimited 1,223 15,784 ms 119.7 ms
BF16 2 (TP=2) 1/s 253 104 ms 16.6 ms
BF16 2 (TP=2) 4/s 981 110 ms 22.0 ms
BF16 2 (TP=2) 16/s 2,141 1,237 ms 72.0 ms
BF16 2 (TP=2) unlimited 2,937 8,320 ms 47.3 ms

How to read this: one GPU running this model serves 36B with 195k tokens of KV cache. Up to 4 requests/s it matches BF16 throughput. It saturates at about 1,230 tok/s, and BF16 reaches 2,937 tok/s on two GPUs with tensor parallelism. The setups differ in GPU count and parallelism, so the gap is not a pure quantization effect. W4A16 kernels reduce weight memory traffic, not compute, so their advantage shrinks as batches grow. Choose this model to cut GPU count or fit smaller GPUs. Choose BF16 on more GPUs for lowest latency under heavy load.

The two benchmarks ran at the same time, on separate GPUs but on shared nodes, with the clients on the GPU nodes. These saturated numbers are lower than in our earlier runs of a 32B model on the same hardware. Treat them as a lower bound and repeat in isolation before relying on them.

Usage

vLLM (recommended)

vllm serve ronantakizawa/seed-oss-36b-base-gptq --max-model-len 8192
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# Base model: use plain completions, not chat
response = client.completions.create(
    model="ronantakizawa/seed-oss-36b-base-gptq",
    prompt="The three most important ideas in distributed systems are",
    max_tokens=200,
)
print(response.choices[0].text)

If the weights sit on a network filesystem or an S3 mount, add --safetensors-load-strategy=eager. On a Mountpoint for S3 volume this cut load time from 190 s to 37 s, because vLLM then reads each file sequentially instead of through random mmap reads.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ronantakizawa/seed-oss-36b-base-gptq"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

  • vLLM, or Transformers ≥ 4.56 (for SeedOssForCausalLM) with compressed-tensors installed (pip install compressed-tensors)
  • ~24 GB+ GPU VRAM (weights ~20 GiB, plus KV cache); 32 GB+ for useful context lengths
  • NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)

License

Apache 2.0 (same as base model)

Citation

@misc{seed-oss-36b-base-gptq,
  title={Seed-OSS-36B-Base GPTQ 4-bit (W4A16)},
  author={Quantized by ronantakizawa},
  year={2026},
  url={https://huggingface.co/ronantakizawa/seed-oss-36b-base-gptq}
}
Downloads last month
27
Safetensors
Model size
6B params
Tensor type
BF16
·
I32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ronantakizawa/seed-oss-36b-base-gptq

Quantized
(11)
this model