Notification Commitment

Nota processes the information below to manage access to this repository and the Notification Commitment.

This repository contains a version of the Base Model that has been compressed and optimized by Nota Inc. ("Nota").

By requesting access, you agree that, if you or your affiliates meet any of the following criteria, you will notify Nota at [contact@nota.ai] before engaging in the applicable activity described below:

(i) your consolidated annual revenue, together with that of your affiliates, exceeds USD 100,000,000, and you intend to use this model for a commercial purpose;

(ii) you design, manufacture, or sell semiconductors, AI accelerators, NPUs, SoCs, or hardware incorporating any of the foregoing, and you intend to distribute this model in connection with such products; or

(iii) you intend to offer this model, or a model derived from it, as part of a paid product, SDK, or hosted service.

This notification commitment is a separate contractual undertaking associated solely with your request for access to this repository.

Nothing contained in this Model Card should be interpreted as or deemed a restriction, modification, or supersession of any applicable license governing this model.

Log in or Sign Up to review the conditions and access this model content.

Kimi-K3.1-Pro-Global-Pruned-37

This model applies Nota AI's non-uniform expert pruning (37.5% of routed experts removed) to moonshotai/Kimi-K3.1-Pro.

1M context on 8 x B200 (TP8) with a 948 GiB checkpoint, FP8 KV cache and DSpark speculative decoding
The original MXFP4 checkpoint (1,454 GiB) does not fit in 8 x B200 memory; the vLLM recipe serves it on 8 x B300. 1M uses --gpu-memory-utilization 0.985; all other arguments follow the vLLM recipe.

  • Routed experts: 82,432 → 51,520 (536–616 per layer; the first two and last two MoE layers keep all 896; layer 0 is dense)
  • Experts per layer: text_config.num_experts_per_layer in config.json; kept expert ids: pruning_info.json
  • Checkpoint size: 1,454 GiB → 948 GiB
  • With the vLLM recipe settings plus FP8 KV cache (max-model-len 1,048,576), this model serves on NVIDIA B200 × 8 (TP8) with a KV cache of 1,415,974 tokens, speculative decoding and the vision encoder enabled, while the original checkpoint does not fit on B200 × 8
  • Not supported with this checkpoint: expert parallel load balancing (EPLB) and the latency_specialized decode path

Installation

Kimi-K3.1-Pro support is not in a vLLM release yet; use the vllm/vllm-openai:kimi-k31 image (vLLM 0.1.dev24216+g266e88c05) and patch the vLLM package inside it.

cd patch
python apply_patch.py --dry-run     # show the changes
python apply_patch.py               # apply the patch (originals are backed up to vllm/.nota_patch_backup/)
python apply_patch.py --revert      # restore the original files
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve <this repo> \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3 \
  --speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic"}' \
  --kv-cache-dtype fp8 \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.985 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 64 \
  --disable-flashinfer-autotune

Performance (vs. the original)

Benchmark Original (model card) Ours (B200 × 8) Δ
IFEval (English) - 95.22 -
IFBench (English) - 76.51 -
GPQA-Diamond (English) 94.20 93.43 -0.77
AIME 2026 (English) - 96.67 -
LiveCodeBench v6 (English) - 92.37 -
BenchMAX IFEval-ko (Korean) - 94.24 -
KoBALT (Korean) - 75.43 -
CLIcK (Korean) - 84.76 -
  • Max generation length: 65,536 tokens (including reasoning); reasoning effort max, temperature 1.0, top-p 0.95 (the settings of the Kimi-K3.1-Pro model card)
  • Original: score reported in the Kimi-K3.1-Pro model card; the other benchmarks are not reported there, and the original does not fit on B200 × 8 for our own evaluation

Notification Commitment

The full Notification Commitment you agree to when requesting access to this repository is in NOTIFICATION_COMMITMENT.md. If any of its criteria apply to you, notify Nota at contact@nota.ai in advance. This commitment does not restrict, modify, or supersede the license below.

License

This model is licensed under the Kimi K3.1-Pro License (LICENSE).

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.

Downloads last month
35
Safetensors
Model size
1.8T params
Tensor type
BF16
·
F32
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nota-ai/Kimi-K3.1-Pro-Global-Pruned-37

Finetuned
(21)
this model

Collection including nota-ai/Kimi-K3.1-Pro-Global-Pruned-37