raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit

This model is a Ternary ({-γ, 0, +γ} packed in 2-bit MLX) mixed-precision quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (converted from mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-bf16) for Apple Silicon using ternary-nemo.

Highlights

  • 6.5x Smaller Footprint: Compressed from 63.16 GB (bfloat16) down to 9.71 GB, running at ~110 tokens/sec in 10.5 GB active Metal RAM on a single Apple Silicon Mac.
  • Strict Symmetric Ternary Experts ({-γ, 0, +γ}): All 23 Mixture-of-Experts layers (128 routed experts per layer, 29.37B parameters / 93.0% of the model) are projected onto per-group symmetric ternary states {-γ, 0, +γ} (group_size=64, mode=ternary-lloyd) and packed into MLX's hardware-accelerated bits=2 QuantizedSwitchLinear format (scale = γ, bias = -γ, q ∈ {0, 1, 2}, ~45–49% exact zero weights).
  • Protected State-Space & Routing Backbone:
    • Mamba-2 in_proj/out_proj, GQA Attention q/k/v/o_proj, Shared Experts, Embeddings, and lm_head (~2.13B params): 4-bit affine (group_size=64).
    • MoE Router Gates (gate.weight, e_score_correction_bias), Mamba-2 conv1d, A_log, D, dt_bias, and RMSNorms: Kept in native bfloat16 / float32.

Quickstart with mlx-lm

No custom kernels or forks are required—this checkpoint runs directly with standard mlx-lm:

pip install -U mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit")

prompt = "Explain why Mixture-of-Experts models are ideal for ternary quantization."
if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=False
    )

response = generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True)
print(response)

Quantization Breakdown

Subsystem Parameters Precision Group Size
Routed MoE Experts (*.mixer.switch_mlp.fc1, fc2) 29.37 B (93.0%) Ternary {-γ, 0, +γ} (2-bit MLX) 64
Shared Experts + Mamba-2 & Attention Projections 1.43 B (4.5%) 4-bit MLX 64
Token Embeddings + lm_head 0.70 B (2.2%) 4-bit MLX 64
MoE Routers, SSM States (A_log, D, dt_bias), conv1d, Norms 0.08 B (0.3%) bfloat16 / float32 —
Total 31.58 B 9.71 GB total —

Zero-Shot PTQ Benchmark Evaluation (Apple Silicon MLX)

We evaluated this 9.71 GB Ternary 2-bit checkpoint across three core benchmarks from NVIDIA's official Nemotron-3.5-Lightning-30B-A3B-BF16 release suite (GSM8K, MMLU-Pro, and GPQA Diamond). All evaluations were performed 100% training-free (zero calibration data / zero QAT) on a single 48 GB Apple Silicon Mac using mlx_lm.batch_generate (enable_thinking=True, max_tokens=2048, batch_size=8):

Benchmark Dataset Evaluated Samples Ternary 2-bit (9.71 GB) NVIDIA BF16 Reference (63.16 GB) Relative Retention
GSM8K (Math Reasoning) openai/gsm8k 200 68.00% (136/200) 95.00% 71.6%
MMLU-Pro (14-Category Stratified) TIGER-Lab/MMLU-Pro 200 42.00% (84/200) 81.94% 51.3%
GPQA Diamond (PhD-Level STEM, no tools) fingertap/GPQA-Diamond 198 (100%) 20.20% (40/198) 75.44% 26.8%

Benchmark Analysis & Key Findings

  1. GSM8K (openai/gsm8k — 68.00% accuracy, 71.6% retention):
    • Even after projecting 93% of the model's parameters (29.37B routed MoE expert weights) onto strict 3-value ternary states ${- \gamma, 0, +\gamma}$ without any fine-tuning or calibration data, the model retains strong multi-step chain-of-thought (<think>...</think>) arithmetic and algebraic reasoning, solving 136 out of 200 test problems.
  2. MMLU-Pro (TIGER-Lab/MMLU-Pro — 42.00% accuracy, 51.3% retention):
    • Evaluated on a deterministic 200-question stratified sample spanning all 14 college-level disciplines (Math, Physics, Chemistry, Law, Engineering, Economics, Health, Computer Science, Biology, Business, Psychology, Philosophy, History, and Other). Unlike standard 4-choice MMLU (25% random baseline), MMLU-Pro is a challenging 10-choice benchmark (10% random baseline), where the ternary model achieves 4.2x random chance.
  3. GPQA Diamond (fingertap/GPQA-Diamond — 20.20% accuracy across all 198 PhD questions):
    • Evaluates all 198 graduate-level physics, chemistry, and biology questions without external tools. Under a strict max_tokens=2048 reasoning budget, roughly half of the PhD-level derivations exceed 2,048 thinking tokens before emitting a final option letter (whereas NVIDIA's full BF16 harness uses max_tokens=16000). On questions where the <think> trace completes within 2,048 tokens, accuracy reaches ~38–40%.
  4. Hardware Efficiency:
    • Single-Stream Inference Speed: 110.1 tokens/sec generation (10.43 GB active Metal memory).
    • Batched Evaluation (batch_size=8): 32.32 GB peak Metal memory on a 48 GB Mac.
    • Quantization Time: 57.5 seconds end-to-end (9.50 GB peak Metal RAM during streaming conversion).

Full per-sample evaluation logs are included in this repository inside benchmark_results.json.

License & Attribution

This checkpoint is a derivative quantized work of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 and is governed by the OpenMDW-1.1 Model License.

Downloads last month
442
Safetensors
Model size
32B params
Tensor type
U32
·
BF16
·
F32
·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit

Datasets used to train raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit

Evaluation results