Instructions to use ronantakizawa/seed-oss-36b-base-gptq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ronantakizawa/seed-oss-36b-base-gptq with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ronantakizawa/seed-oss-36b-base-gptq") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("ronantakizawa/seed-oss-36b-base-gptq") model = AutoModelForCausalLM.from_pretrained("ronantakizawa/seed-oss-36b-base-gptq", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ronantakizawa/seed-oss-36b-base-gptq with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ronantakizawa/seed-oss-36b-base-gptq" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ronantakizawa/seed-oss-36b-base-gptq", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ronantakizawa/seed-oss-36b-base-gptq
- SGLang
How to use ronantakizawa/seed-oss-36b-base-gptq with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ronantakizawa/seed-oss-36b-base-gptq" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ronantakizawa/seed-oss-36b-base-gptq", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ronantakizawa/seed-oss-36b-base-gptq" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ronantakizawa/seed-oss-36b-base-gptq", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use ronantakizawa/seed-oss-36b-base-gptq with Docker Model Runner:
docker model run hf.co/ronantakizawa/seed-oss-36b-base-gptq
Seed-OSS-36B-Base GPTQ 4-bit (W4A16)
This is a 4-bit GPTQ quantized version of ByteDance-Seed/Seed-OSS-36B-Base using LLM Compressor, saved in the compressed-tensors format that vLLM loads natively.
Key Features
- 36B parameters quantized to 4-bit - 71% size reduction (72.3 GB → 21.0 GB)
- Fits on a single GPU - weights use 19.9 GiB in vLLM, instead of needing 2× 80 GB GPUs in BF16
- Every decoder Linear layer quantized with GPTQ, including attention
- Apache 2.0 base model with a 512K-token context window
- vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
- +1.22% perplexity vs BF16 on C4 validation; ARC-Challenge 61.4% vs 61.1%, HellaSwag 81.6% vs 82.3%, GSM8K 83.0% vs 86.7%
- Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes
Model Details
- Base Model: ByteDance-Seed/Seed-OSS-36B-Base (36.2B parameters, base/pretrained model, not instruction-tuned; this is the version trained with synthetic instruction data, not the
-woSynvariant) - Architecture: Seed-OSS (
SeedOssForCausalLM), 64 layers, hidden size 5120, grouped-query attention (80 query heads, 8 KV heads, head dim 128), MLP size 27,648, vocabulary 155,136, biases on the q/k/v projections - Quantization Method: GPTQ
- Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
- Group Size: 128, symmetric, static activation ordering, dampening 0.01
- Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)
Size Comparison
| Metric | Value |
|---|---|
| Original (BF16) | ~72.3 GB |
| Quantized (W4A16) | ~21.0 GB |
| Reduction | ~71% |
| Memory Saved | ~51.3 GB |
The quantized share is smaller than for models with smaller vocabularies: the 155K-token embedding table and lm_head stay in BF16 (3.2 GB of the 21.0 GB).
What Was Quantized
Quantized (4-bit):
- All 448 decoder Linear layers:
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_projin all 64 layers
Preserved (BF16):
lm_head- Token embeddings
- Normalization layers
- Attention projection biases
How It Was Quantized: Multi-Node Distributed GPTQ
This model was quantized on Kubernetes (Amazon EKS, us-west-2) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:
| Step | Detail |
|---|---|
| Launch | Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; torchrun with 8 processes per node (16 ranks) |
| Interconnect | NVLink inside each node; EFA between nodes via aws-ofi-nccl |
| Storage | BF16 weights and calibration data read from an S3 bucket mounted as a Kubernetes PersistentVolume (Mountpoint for Amazon S3 CSI driver); the checkpoint was written to a second S3-backed PersistentVolume. No pod downloaded the model |
| Model loading | Rank 0 reads the BF16 weights from the volume and broadcasts replicas to all 16 ranks over NCCL (240 s) |
| Calibration | Each rank runs forward passes on its own 16-document slice (data-parallel calibration) |
| Compression | Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points |
| Runtime | 179 s for all 65 sequential stages; peak GPU memory 77.1 GiB (highest across ranks) |
Notes from the run:
- Memory. Every rank holds a full BF16 replica (67.3 GiB), which leaves little room for GPTQ on an 80 GB H100. The embedding table and
lm_head(3.0 GiB, untouched by GPTQ) were moved to each rank's own CPU memory during calibration and moved back to the GPU for saving. Peak was 77.1 GiB. - Multi-node device bug. LLM Compressor 0.14.0's
get_main_device()selects the GPU by global rank, which fails on the second node (cuda:8–cuda:15don't exist on an 8-GPU machine). The run patches it to use the local device index. - Offload across nodes. LLM Compressor rebuilds offloading around
save_pretrained. Modules left on CPU get a cache shared through/dev/shm, which ranks on another node can't open. Keeping every module on the GPU before saving avoids it.
Evaluation
Tested with vLLM 0.30 on p5.48xlarge nodes (H100 80 GB): this model on 1 GPU, the BF16 base model on 2 GPUs (tensor parallel) as the baseline. vLLM loads this checkpoint with MacheteLinearKernel (compressed-tensors WNA16); weights use 19.85 GiB of GPU memory.
Perplexity
C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):
| Model | Perplexity |
|---|---|
| BF16 (original) | 9.010 |
| GPTQ W4A16 (this model) | 9.120 (+1.22%) |
Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16. A Fibonacci function prompt produced the same function body.
Task Accuracy
lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):
| Task | Metric | BF16 | GPTQ W4A16 | Difference |
|---|---|---|---|---|
| ARC-Challenge (0-shot) | acc_norm | 61.09% | 61.43% | +0.34 |
| HellaSwag (0-shot) | acc_norm | 82.28% | 81.60% | −0.68 |
| GSM8K (5-shot) | exact_match, flexible-extract | 86.66% | 83.02% | −3.64 |
| GSM8K (5-shot) | exact_match, strict | 76.65% | 63.61% | −13.04 |
Next-Token Agreement
The same 102,300 C4 validation positions, top-20 log-probabilities from each server:
| Measure | Value |
|---|---|
| Same top-1 token as BF16 | 94.39% |
| BF16's top-1 token within this model's top-5 | 99.94% |
| KL(BF16 ‖ GPTQ), mean / median / p99 | 0.0144 / 0.0073 / 0.129 nats |
The KL estimate uses only BF16's top-20 tokens (88.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.
Throughput and Latency
vllm bench serve, random 1,024-token inputs / 256-token outputs, 400 prompts, --max-model-len 8192:
| Model | GPUs | Request rate | Output tok/s | Median TTFT | Median TPOT |
|---|---|---|---|---|---|
| GPTQ W4A16 | 1 | 1/s | 254 | 164 ms | 15.3 ms |
| GPTQ W4A16 | 1 | 4/s | 982 | 271 ms | 38.9 ms |
| GPTQ W4A16 | 1 | 16/s | 1,230 | 7,564 ms | 119.8 ms |
| GPTQ W4A16 | 1 | unlimited | 1,223 | 15,784 ms | 119.7 ms |
| BF16 | 2 (TP=2) | 1/s | 253 | 104 ms | 16.6 ms |
| BF16 | 2 (TP=2) | 4/s | 981 | 110 ms | 22.0 ms |
| BF16 | 2 (TP=2) | 16/s | 2,141 | 1,237 ms | 72.0 ms |
| BF16 | 2 (TP=2) | unlimited | 2,937 | 8,320 ms | 47.3 ms |
How to read this: one GPU running this model serves 36B with 195k tokens of KV cache. Up to 4 requests/s it matches BF16 throughput. It saturates at about 1,230 tok/s, and BF16 reaches 2,937 tok/s on two GPUs with tensor parallelism. The setups differ in GPU count and parallelism, so the gap is not a pure quantization effect. W4A16 kernels reduce weight memory traffic, not compute, so their advantage shrinks as batches grow. Choose this model to cut GPU count or fit smaller GPUs. Choose BF16 on more GPUs for lowest latency under heavy load.
The two benchmarks ran at the same time, on separate GPUs but on shared nodes, with the clients on the GPU nodes. These saturated numbers are lower than in our earlier runs of a 32B model on the same hardware. Treat them as a lower bound and repeat in isolation before relying on them.
Usage
vLLM (recommended)
vllm serve ronantakizawa/seed-oss-36b-base-gptq --max-model-len 8192
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# Base model: use plain completions, not chat
response = client.completions.create(
model="ronantakizawa/seed-oss-36b-base-gptq",
prompt="The three most important ideas in distributed systems are",
max_tokens=200,
)
print(response.choices[0].text)
If the weights sit on a network filesystem or an S3 mount, add --safetensors-load-strategy=eager. On a Mountpoint for S3 volume this cut load time from 190 s to 37 s, because vLLM then reads each file sequentially instead of through random mmap reads.
Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ronantakizawa/seed-oss-36b-base-gptq"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Requirements
- vLLM, or Transformers ≥ 4.56 (for
SeedOssForCausalLM) withcompressed-tensorsinstalled (pip install compressed-tensors) - ~24 GB+ GPU VRAM (weights ~20 GiB, plus KV cache); 32 GB+ for useful context lengths
- NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)
License
Apache 2.0 (same as base model)
Citation
@misc{seed-oss-36b-base-gptq,
title={Seed-OSS-36B-Base GPTQ 4-bit (W4A16)},
author={Quantized by ronantakizawa},
year={2026},
url={https://huggingface.co/ronantakizawa/seed-oss-36b-base-gptq}
}
- Downloads last month
- 27
Model tree for ronantakizawa/seed-oss-36b-base-gptq
Base model
ByteDance-Seed/Seed-OSS-36B-Base