K2-Horizon-MoVA-36B-A4B-MLA-oQ6e

Experimental. This is not the standard IFM/K2-Horizon-MoVA-36B-A4B. Its attention is changed, and its output is different from the original model's output (see Quality). For normal work, use the original model or a quantization of the original. Use this model to try very long contexts (up to 512K tokens) on a computer with less memory.

K2-Horizon-MoVA-36B-A4B-MLA-oQ6e is an experiment based on IFM/K2-Horizon-MoVA-36B-A4B. Most of its attention layers use multi-head latent attention (MLA) instead of grouped-query attention (GQA). Thus its KV cache is 45% smaller than the original's in BF16, and 61% smaller with the default q8 cache. With these weights and the rq4 cache, 512K tokens of context need about 67 GB in total (a 96 GB Mac or larger; see KV cache size).

Weights: oMLX oQe quantization (imatrix calibration, mixed precision), about 6 bits per weight.

How it is different from the original model

The original model uses grouped-query attention (GQA) in all 48 layers. Each layer keeps 8 key heads and 8 value heads of 128 values for each token: 2,048 values per layer.

This model is a hybrid:

  • 38 layers use MLA. Each of these layers keeps one latent vector of 768 values and one RoPE key of 128 values for each token: 896 values per layer. The attention computes the keys and values of the 32 query heads from the latent.
  • 10 layers keep GQA (layers 32, 35, 37, 39, 41, 42, 43, 45, 46 and 47). In these layers, MLA changes the output most. Thus these layers keep their original attention and their full cache.

The conversion follows TransMLA (arXiv:2502.07864), with calibration on text. Distillation from the original model then trained the MLA projections (healing). The routed experts, the value experts, the routers and the other weights are the original weights (in this repository's encoding).

KV cache size

Cache format Per token 32K tokens 128K tokens 200K tokens 512K tokens
Original GQA model, BF16 192 KiB 6.4 GB 25.8 GB 39.3 GB 103.1 GB
This model, BF16 106.5 KiB 3.6 GB 14.3 GB 21.8 GB 57.2 GB
This model, q8 (default) 75.3 KiB 2.5 GB 10.1 GB 15.4 GB 40.4 GB
This model, rq4 61.1 KiB 2.0 GB 8.2 GB 12.5 GB 32.8 GB
  • q8: the MLA latent and RoPE key in 8 bits. This is the default. Its effect on quality is very small (KL divergence +0.0004 against the BF16 cache).
  • rq4: the MLA latent in 4 bits after a seeded Walsh-Hadamard rotation, the RoPE key in 8 bits. The model file selects rq4 when the context that you ask for does not fit as q8. Its effect on quality is small (KL divergence +0.004).
  • The 10 GQA layers keep BF16 keys and values in all formats. They are 40 KiB of each token's cache.

Memory for oQ6e: 31.6 GB of weights. With 128K tokens in the q8 cache, about 44 GB in total. With 512K tokens in the rq4 cache, about 67 GB in total. (Totals include 2.5 GB for prompt processing.)

Longest context by unified memory, with macOS's default GPU memory limit (three quarters of memory). 512K tokens is the model's maximum context:

Unified memory BF16 cache q8 cache rq4 cache
64 GB 156K 220K 272K
96 GB 387K 512K 512K
128 GB 512K 512K 512K
192 GB 512K 512K 512K

Quality

KL divergence from the original GQA model (BF16), on 31 held-out text windows of 1,024 tokens (31,713 positions). Lower is closer to the original model. All six versions use the same healed MLA weights:

Repository Weights Size Bits per weight KL divergence
K2-Horizon-MoVA-36B-A4B-MLA BF16 75.6 GB 16.0 0.105
K2-Horizon-MoVA-36B-A4B-MLA-oQ8e oQe 40.5 GB 8.7 0.104
K2-Horizon-MoVA-36B-A4B-MLA-oQ6e (this model) oQe 31.6 GB 6.8 0.105
K2-Horizon-MoVA-36B-A4B-MLA-oQ4e oQe 22.7 GB 4.9 0.112
K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4 SeedLM seeds 22.2 GB 4.7 0.113
K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P3 SeedLM seeds 20.0 GB 4.3 0.122
K2-Horizon-MoVA-36B-A4B-oQ4e (original GQA attention, for comparison) oQe 22.0 GB 4.8 0.031

Platforms

  • Apple Silicon Macs with oMLX or mlx-lm (MLX). Enable Trust Remote Code: the model file k2_mla.py is in this repository.

How to run

oMLX

  1. Use an oMLX version with K2-Horizon support (mlx_lm.models.k2_horizon).
  2. Download this repository to the oMLX model folder.
  3. In the model settings, enable Trust Remote Code.

mlx-lm

from omlx.patches.k2_horizon import apply_k2_horizon_patch
apply_k2_horizon_patch()            # makes mlx_lm.models.k2_horizon available

from mlx_lm import load, generate
model, tokenizer = load("txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e", trust_remote_code=True)

Long context

To use more context than q8 can hold, tell the model file the longest context that you will use. Do one of these:

  • Set NSLM_MAX_CONTEXT=524288 in the environment of the process that loads the model.
  • Set "mla_max_context": 524288 in config.json.

The model file then selects the cache format. To keep the BF16 cache, set "mla_kv": "bf16" in config.json. Read The KV cache in the NanoSeedLM documentation.

Limits

  • This is an experiment. The model is not identical to the original model, and it is not a replacement for it. The KL divergence in the Quality section shows the difference.
  • You must enable Trust Remote Code. The model file needs the K2-Horizon support of oMLX.
  • Prompt processing is slow at long context. On an M4 Max, at 128K tokens of context, one 2,048-token prompt chunk takes about 45 s. Use prompt-prefix reuse (oMLX keeps prefixes in memory and on SSD) so that each turn processes only its new tokens.
  • oMLX calculates memory for the original GQA cache (192 KiB per token). Thus oMLX can admit less context than this model can hold.
  • In the NanoSeedLM engine, hybrid MLA models need the engine version with hybrid support. Earlier versions do not load them.

Files

  • model-*.safetensors: the weights (oQ6e).
  • k2_mla.py: the model file (MLX).
  • config.json: the configuration, with mla_ranks (768 for an MLA layer, 0 for a GQA layer).

Earlier versions: the tag pre-t03 keeps the all-MLA version (84 KiB of BF16 cache per token, KL divergence about 0.15).

Credits

Base model: IFM, K2-Horizon-MoVA-36B-A4B (Apache-2.0). Methods: TransMLA (Meng et al., arXiv:2502.07864). Code: NanoSeedLM.

Downloads last month
37
Safetensors
Model size
38B params
Tensor type
U32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

6-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e

Quantized
(5)
this model

Collection including txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e

Paper for txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e