Instructions to use nota-ai/Kimi-K3.1-Pro-Global-Pruned-37 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nota-ai/Kimi-K3.1-Pro-Global-Pruned-37 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="nota-ai/Kimi-K3.1-Pro-Global-Pruned-37", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("nota-ai/Kimi-K3.1-Pro-Global-Pruned-37", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use nota-ai/Kimi-K3.1-Pro-Global-Pruned-37 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/nota-ai/Kimi-K3.1-Pro-Global-Pruned-37
- SGLang
How to use nota-ai/Kimi-K3.1-Pro-Global-Pruned-37 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nota-ai/Kimi-K3.1-Pro-Global-Pruned-37", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use nota-ai/Kimi-K3.1-Pro-Global-Pruned-37 with Docker Model Runner:
docker model run hf.co/nota-ai/Kimi-K3.1-Pro-Global-Pruned-37
Notification Commitment
Nota processes the information below to manage access to this repository and the Notification Commitment.
This repository contains a version of the Base Model that has been compressed and optimized by Nota Inc. ("Nota").
By requesting access, you agree that, if you or your affiliates meet any of the following criteria, you will notify Nota at [contact@nota.ai] before engaging in the applicable activity described below:
(i) your consolidated annual revenue, together with that of your affiliates, exceeds USD 100,000,000, and you intend to use this model for a commercial purpose;
(ii) you design, manufacture, or sell semiconductors, AI accelerators, NPUs, SoCs, or hardware incorporating any of the foregoing, and you intend to distribute this model in connection with such products; or
(iii) you intend to offer this model, or a model derived from it, as part of a paid product, SDK, or hosted service.
This notification commitment is a separate contractual undertaking associated solely with your request for access to this repository.
Nothing contained in this Model Card should be interpreted as or deemed a restriction, modification, or supersession of any applicable license governing this model.
Log in or Sign Up to review the conditions and access this model content.
Kimi-K3.1-Pro-Global-Pruned-37
This model applies Nota AI's non-uniform expert pruning (37.5% of routed experts removed) to moonshotai/Kimi-K3.1-Pro.
1M context on 8 x B200 (TP8) with a 948 GiB checkpoint, FP8 KV cache and DSpark speculative decoding
The original MXFP4 checkpoint (1,454 GiB) does not fit in 8 x B200 memory; the vLLM recipe serves it on
8 x B300. 1M uses --gpu-memory-utilization 0.985; all other arguments follow the vLLM recipe.
- Routed experts: 82,432 → 51,520 (536–616 per layer; the first two and last two MoE layers keep all 896; layer 0 is dense)
- Experts per layer:
text_config.num_experts_per_layerinconfig.json; kept expert ids:pruning_info.json - Checkpoint size: 1,454 GiB → 948 GiB
- With the vLLM recipe settings plus FP8 KV cache (max-model-len 1,048,576), this model serves on NVIDIA B200 × 8 (TP8) with a KV cache of 1,415,974 tokens, speculative decoding and the vision encoder enabled, while the original checkpoint does not fit on B200 × 8
- Not supported with this checkpoint: expert parallel load balancing (EPLB) and the
latency_specializeddecode path
Installation
Kimi-K3.1-Pro support is not in a vLLM release yet; use the vllm/vllm-openai:kimi-k31 image
(vLLM 0.1.dev24216+g266e88c05) and patch the vLLM package inside it.
cd patch
python apply_patch.py --dry-run # show the changes
python apply_patch.py # apply the patch (originals are backed up to vllm/.nota_patch_backup/)
python apply_patch.py --revert # restore the original files
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve <this repo> \
--trust-remote-code \
--tensor-parallel-size 8 \
--load-format fastsafetensors \
--enable-auto-tool-choice \
--tool-call-parser kimi_k3 \
--reasoning-parser kimi_k3 \
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic"}' \
--kv-cache-dtype fp8 \
--max-model-len 1048576 \
--gpu-memory-utilization 0.985 \
--max-num-batched-tokens 4096 \
--max-num-seqs 64 \
--disable-flashinfer-autotune
Performance (vs. the original)
| Benchmark | Original (model card) | Ours (B200 × 8) | Δ |
|---|---|---|---|
| IFEval (English) | - | 95.22 | - |
| IFBench (English) | - | 76.51 | - |
| GPQA-Diamond (English) | 94.20 | 93.43 | -0.77 |
| AIME 2026 (English) | - | 96.67 | - |
| LiveCodeBench v6 (English) | - | 92.37 | - |
| BenchMAX IFEval-ko (Korean) | - | 94.24 | - |
| KoBALT (Korean) | - | 75.43 | - |
| CLIcK (Korean) | - | 84.76 | - |
- Max generation length: 65,536 tokens (including reasoning); reasoning effort max, temperature 1.0, top-p 0.95 (the settings of the Kimi-K3.1-Pro model card)
- Original: score reported in the Kimi-K3.1-Pro model card; the other benchmarks are not reported there, and the original does not fit on B200 × 8 for our own evaluation
Notification Commitment
The full Notification Commitment you agree to when requesting access to this repository is in
NOTIFICATION_COMMITMENT.md. If any of its criteria apply to you, notify Nota at
contact@nota.ai in advance. This commitment does not restrict, modify, or supersede the
license below.
License
This model is licensed under the Kimi K3.1-Pro License (LICENSE).
You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.
- Downloads last month
- 35
Model tree for nota-ai/Kimi-K3.1-Pro-Global-Pruned-37
Base model
moonshotai/Kimi-K3.1-Pro