Instructions to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit
This model is a Ternary ({-γ, 0, +γ} packed in 2-bit MLX) mixed-precision quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (converted from mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-bf16) for Apple Silicon using ternary-nemo.
Highlights
- 6.5x Smaller Footprint: Compressed from 63.16 GB (
bfloat16) down to 9.71 GB, running at ~110 tokens/sec in 10.5 GB active Metal RAM on a single Apple Silicon Mac. - Strict Symmetric Ternary Experts (
{-γ, 0, +γ}): All 23 Mixture-of-Experts layers (128routed experts per layer, 29.37B parameters / 93.0% of the model) are projected onto per-group symmetric ternary states{-γ, 0, +γ}(group_size=64, mode=ternary-lloyd) and packed into MLX's hardware-acceleratedbits=2QuantizedSwitchLinearformat (scale = γ,bias = -γ,q ∈ {0, 1, 2}, ~45–49% exact zero weights). - Protected State-Space & Routing Backbone:
- Mamba-2
in_proj/out_proj, GQA Attentionq/k/v/o_proj, Shared Experts, Embeddings, andlm_head(~2.13B params): 4-bit affine (group_size=64). - MoE Router Gates (
gate.weight,e_score_correction_bias), Mamba-2conv1d,A_log,D,dt_bias, and RMSNorms: Kept in nativebfloat16/float32.
- Mamba-2
Quickstart with mlx-lm
No custom kernels or forks are required—this checkpoint runs directly with standard mlx-lm:
pip install -U mlx-lm
from mlx_lm import load, generate
model, tokenizer = load("raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit")
prompt = "Explain why Mixture-of-Experts models are ideal for ternary quantization."
if tokenizer.chat_template is not None:
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, tokenize=False
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True)
print(response)
Quantization Breakdown
| Subsystem | Parameters | Precision | Group Size |
|---|---|---|---|
Routed MoE Experts (*.mixer.switch_mlp.fc1, fc2) |
29.37 B (93.0%) | Ternary {-γ, 0, +γ} (2-bit MLX) |
64 |
| Shared Experts + Mamba-2 & Attention Projections | 1.43 B (4.5%) | 4-bit MLX | 64 |
Token Embeddings + lm_head |
0.70 B (2.2%) | 4-bit MLX | 64 |
MoE Routers, SSM States (A_log, D, dt_bias), conv1d, Norms |
0.08 B (0.3%) | bfloat16 / float32 |
— |
| Total | 31.58 B | 9.71 GB total | — |
Zero-Shot PTQ Benchmark Evaluation (Apple Silicon MLX)
We evaluated this 9.71 GB Ternary 2-bit checkpoint across three core benchmarks from NVIDIA's official Nemotron-3.5-Lightning-30B-A3B-BF16 release suite (GSM8K, MMLU-Pro, and GPQA Diamond). All evaluations were performed 100% training-free (zero calibration data / zero QAT) on a single 48 GB Apple Silicon Mac using mlx_lm.batch_generate (enable_thinking=True, max_tokens=2048, batch_size=8):
| Benchmark | Dataset | Evaluated Samples | Ternary 2-bit (9.71 GB) | NVIDIA BF16 Reference (63.16 GB) | Relative Retention |
|---|---|---|---|---|---|
| GSM8K (Math Reasoning) | openai/gsm8k |
200 | 68.00% (136/200) | 95.00% | 71.6% |
| MMLU-Pro (14-Category Stratified) | TIGER-Lab/MMLU-Pro |
200 | 42.00% (84/200) | 81.94% | 51.3% |
| GPQA Diamond (PhD-Level STEM, no tools) | fingertap/GPQA-Diamond |
198 (100%) | 20.20% (40/198) | 75.44% | 26.8% |
Benchmark Analysis & Key Findings
- GSM8K (
openai/gsm8k— 68.00% accuracy, 71.6% retention):- Even after projecting 93% of the model's parameters (
29.37Brouted MoE expert weights) onto strict 3-value ternary states ${- \gamma, 0, +\gamma}$ without any fine-tuning or calibration data, the model retains strong multi-step chain-of-thought (<think>...</think>) arithmetic and algebraic reasoning, solving 136 out of 200 test problems.
- Even after projecting 93% of the model's parameters (
- MMLU-Pro (
TIGER-Lab/MMLU-Pro— 42.00% accuracy, 51.3% retention):- Evaluated on a deterministic 200-question stratified sample spanning all 14 college-level disciplines (Math, Physics, Chemistry, Law, Engineering, Economics, Health, Computer Science, Biology, Business, Psychology, Philosophy, History, and Other). Unlike standard 4-choice MMLU (25% random baseline), MMLU-Pro is a challenging 10-choice benchmark (10% random baseline), where the ternary model achieves 4.2x random chance.
- GPQA Diamond (
fingertap/GPQA-Diamond— 20.20% accuracy across all 198 PhD questions):- Evaluates all 198 graduate-level physics, chemistry, and biology questions without external tools. Under a strict
max_tokens=2048reasoning budget, roughly half of the PhD-level derivations exceed 2,048 thinking tokens before emitting a final option letter (whereas NVIDIA's full BF16 harness usesmax_tokens=16000). On questions where the<think>trace completes within 2,048 tokens, accuracy reaches ~38–40%.
- Evaluates all 198 graduate-level physics, chemistry, and biology questions without external tools. Under a strict
- Hardware Efficiency:
- Single-Stream Inference Speed: 110.1 tokens/sec generation (
10.43 GBactive Metal memory). - Batched Evaluation (
batch_size=8): 32.32 GB peak Metal memory on a 48 GB Mac. - Quantization Time: 57.5 seconds end-to-end (
9.50 GBpeak Metal RAM during streaming conversion).
- Single-Stream Inference Speed: 110.1 tokens/sec generation (
Full per-sample evaluation logs are included in this repository inside benchmark_results.json.
License & Attribution
This checkpoint is a derivative quantized work of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 and is governed by the OpenMDW-1.1 Model License.
- Downloads last month
- 442
4-bit
Model tree for raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit
Datasets used to train raghul2004/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Ternary-2bit
TIGER-Lab/MMLU-Pro
fingertap/GPQA-Diamond
Evaluation results
- Accuracy (Zero-Shot Ternary PTQ) on GSM8Ktest set self-reported68.000
- Accuracy (14-Category Stratified) on MMLU-Protest set self-reported42.000
- Accuracy (No Tools, 2048 Token Budget) on GPQA Diamondtest set self-reported20.200