Instructions to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
K2-Horizon-MoVA-36B-A4B-MLA-oQ6e
Experimental. This is not the standard IFM/K2-Horizon-MoVA-36B-A4B. Its attention is changed, and its output is different from the original model's output (see Quality). For normal work, use the original model or a quantization of the original. Use this model to try very long contexts (up to 512K tokens) on a computer with less memory.
K2-Horizon-MoVA-36B-A4B-MLA-oQ6e is an experiment based on IFM/K2-Horizon-MoVA-36B-A4B. Most of its attention layers use multi-head latent attention (MLA) instead of grouped-query attention (GQA). Thus its KV cache is 45% smaller than the original's in BF16, and 61% smaller with the default q8 cache. With these weights and the rq4 cache, 512K tokens of context need about 67 GB in total (a 96 GB Mac or larger; see KV cache size).
Weights: oMLX oQe quantization (imatrix calibration, mixed precision), about 6 bits per weight.
How it is different from the original model
The original model uses grouped-query attention (GQA) in all 48 layers. Each layer keeps 8 key heads and 8 value heads of 128 values for each token: 2,048 values per layer.
This model is a hybrid:
- 38 layers use MLA. Each of these layers keeps one latent vector of 768 values and one RoPE key of 128 values for each token: 896 values per layer. The attention computes the keys and values of the 32 query heads from the latent.
- 10 layers keep GQA (layers 32, 35, 37, 39, 41, 42, 43, 45, 46 and 47). In these layers, MLA changes the output most. Thus these layers keep their original attention and their full cache.
The conversion follows TransMLA (arXiv:2502.07864), with calibration on text. Distillation from the original model then trained the MLA projections (healing). The routed experts, the value experts, the routers and the other weights are the original weights (in this repository's encoding).
KV cache size
| Cache format | Per token | 32K tokens | 128K tokens | 200K tokens | 512K tokens |
|---|---|---|---|---|---|
| Original GQA model, BF16 | 192 KiB | 6.4 GB | 25.8 GB | 39.3 GB | 103.1 GB |
| This model, BF16 | 106.5 KiB | 3.6 GB | 14.3 GB | 21.8 GB | 57.2 GB |
| This model, q8 (default) | 75.3 KiB | 2.5 GB | 10.1 GB | 15.4 GB | 40.4 GB |
| This model, rq4 | 61.1 KiB | 2.0 GB | 8.2 GB | 12.5 GB | 32.8 GB |
- q8: the MLA latent and RoPE key in 8 bits. This is the default. Its effect on quality is very small (KL divergence +0.0004 against the BF16 cache).
- rq4: the MLA latent in 4 bits after a seeded Walsh-Hadamard rotation, the RoPE key in 8 bits. The model file selects rq4 when the context that you ask for does not fit as q8. Its effect on quality is small (KL divergence +0.004).
- The 10 GQA layers keep BF16 keys and values in all formats. They are 40 KiB of each token's cache.
Memory for oQ6e: 31.6 GB of weights. With 128K tokens in the q8 cache, about 44 GB in total. With 512K tokens in the rq4 cache, about 67 GB in total. (Totals include 2.5 GB for prompt processing.)
Longest context by unified memory, with macOS's default GPU memory limit (three quarters of memory). 512K tokens is the model's maximum context:
| Unified memory | BF16 cache | q8 cache | rq4 cache |
|---|---|---|---|
| 64 GB | 156K | 220K | 272K |
| 96 GB | 387K | 512K | 512K |
| 128 GB | 512K | 512K | 512K |
| 192 GB | 512K | 512K | 512K |
Quality
KL divergence from the original GQA model (BF16), on 31 held-out text windows of 1,024 tokens (31,713 positions). Lower is closer to the original model. All six versions use the same healed MLA weights:
| Repository | Weights | Size | Bits per weight | KL divergence |
|---|---|---|---|---|
| K2-Horizon-MoVA-36B-A4B-MLA | BF16 | 75.6 GB | 16.0 | 0.105 |
| K2-Horizon-MoVA-36B-A4B-MLA-oQ8e | oQe | 40.5 GB | 8.7 | 0.104 |
| K2-Horizon-MoVA-36B-A4B-MLA-oQ6e (this model) | oQe | 31.6 GB | 6.8 | 0.105 |
| K2-Horizon-MoVA-36B-A4B-MLA-oQ4e | oQe | 22.7 GB | 4.9 | 0.112 |
| K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4 | SeedLM seeds | 22.2 GB | 4.7 | 0.113 |
| K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P3 | SeedLM seeds | 20.0 GB | 4.3 | 0.122 |
| K2-Horizon-MoVA-36B-A4B-oQ4e (original GQA attention, for comparison) | oQe | 22.0 GB | 4.8 | 0.031 |
Platforms
- Apple Silicon Macs with oMLX or mlx-lm (MLX). Enable Trust Remote Code: the
model file
k2_mla.pyis in this repository.
How to run
oMLX
- Use an oMLX version with K2-Horizon support (
mlx_lm.models.k2_horizon). - Download this repository to the oMLX model folder.
- In the model settings, enable Trust Remote Code.
mlx-lm
from omlx.patches.k2_horizon import apply_k2_horizon_patch
apply_k2_horizon_patch() # makes mlx_lm.models.k2_horizon available
from mlx_lm import load, generate
model, tokenizer = load("txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e", trust_remote_code=True)
Long context
To use more context than q8 can hold, tell the model file the longest context that you will use. Do one of these:
- Set
NSLM_MAX_CONTEXT=524288in the environment of the process that loads the model. - Set
"mla_max_context": 524288inconfig.json.
The model file then selects the cache format. To keep the BF16 cache, set "mla_kv": "bf16" in config.json. Read
The KV cache in the NanoSeedLM documentation.
Limits
- This is an experiment. The model is not identical to the original model, and it is not a replacement for it. The KL divergence in the Quality section shows the difference.
- You must enable Trust Remote Code. The model file needs the K2-Horizon support of oMLX.
- Prompt processing is slow at long context. On an M4 Max, at 128K tokens of context, one 2,048-token prompt chunk takes about 45 s. Use prompt-prefix reuse (oMLX keeps prefixes in memory and on SSD) so that each turn processes only its new tokens.
- oMLX calculates memory for the original GQA cache (192 KiB per token). Thus oMLX can admit less context than this model can hold.
- In the NanoSeedLM engine, hybrid MLA models need the engine version with hybrid support. Earlier versions do not load them.
Files
model-*.safetensors: the weights (oQ6e).k2_mla.py: the model file (MLX).config.json: the configuration, withmla_ranks(768 for an MLA layer, 0 for a GQA layer).
Earlier versions: the tag pre-t03 keeps the all-MLA version (84 KiB of BF16 cache per token, KL divergence about
0.15).
Credits
Base model: IFM, K2-Horizon-MoVA-36B-A4B (Apache-2.0). Methods: TransMLA (Meng et al., arXiv:2502.07864). Code: NanoSeedLM.
- Downloads last month
- 37
6-bit
Model tree for txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e
Base model
IFM/K2-Horizon-MoVA-36B-A4B