Text Generation
MLX
Safetensors
Arabic
English
k2_horizon
jais-3
Mixture of Experts
quantized
mova
mixture-of-experts
conversational
4-bit precision
Instructions to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("IFM/Jais-3-MoVA-36B-A4B-MLX-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default IFM/Jais-3-MoVA-36B-A4B-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use IFM/Jais-3-MoVA-36B-A4B-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "IFM/Jais-3-MoVA-36B-A4B-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Upload MLX conversion
Browse files- .gitattributes +1 -0
- README.md +98 -0
- chat_template.jinja +994 -0
- config.json +0 -0
- generation_config.json +8 -0
- k2_horizon.py +628 -0
- model-00001-of-00005.safetensors +3 -0
- model-00002-of-00005.safetensors +3 -0
- model-00003-of-00005.safetensors +3 -0
- model-00004-of-00005.safetensors +3 -0
- model-00005-of-00005.safetensors +3 -0
- model.safetensors.index.json +0 -0
- tokenizer.json +3 -0
- tokenizer_config.json +10 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
license_link: https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B
|
| 4 |
+
base_model: IFM/Jais-3-MoVA-36B-A4B
|
| 5 |
+
base_model_relation: quantized
|
| 6 |
+
library_name: mlx
|
| 7 |
+
pipeline_tag: text-generation
|
| 8 |
+
language:
|
| 9 |
+
- ar
|
| 10 |
+
- en
|
| 11 |
+
tags:
|
| 12 |
+
- mlx
|
| 13 |
+
- jais-3
|
| 14 |
+
- moe
|
| 15 |
+
- quantized
|
| 16 |
+
- mova
|
| 17 |
+
- mixture-of-experts
|
| 18 |
+
---
|
| 19 |
+
# Jais-3-MoVA-36B-A4B (MLX, 4-bit)
|
| 20 |
+
|
| 21 |
+
Mixed-precision **4-bit** MLX quantization of [IFM/Jais-3-MoVA-36B-A4B](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B), converted from revision `50e83d4`. **5.6 bits/weight effective**, 26.4 GB on disk. For Apple silicon.
|
| 22 |
+
|
| 23 |
+
Jais-3-MoVA-36B-A4B is a mixture-of-experts Jais-3 model with Mixture-of-Values (MoVA) attention: 36B total parameters, about 4B active per token, 512K context window.
|
| 24 |
+
|
| 25 |
+
## Requirements
|
| 26 |
+
|
| 27 |
+
mlx-lm doesn't support the `k2_horizon` architecture (used by Jais-3) yet. Until support lands, **this repo ships the MLX model code** ([`k2_horizon.py`](./k2_horizon.py)), which mlx-lm loads through the `model_file` entry in `config.json`. So the released mlx-lm works as-is:
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
pip install -U mlx-lm
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
Pass `--trust-remote-code` (or `trust_remote_code=True`). The file runs on your machine, so read it first if you like; its comments describe how it differs from the PyTorch code.
|
| 34 |
+
|
| 35 |
+
## How it was quantized
|
| 36 |
+
|
| 37 |
+
Mixed precision, same recipe as the K2-Horizon MoVA MLX model (integer affine, bf16 scales and biases):
|
| 38 |
+
|
| 39 |
+
- **4-bit, group size 64**: the routed MoE experts and the shared expert. These hold most of the parameters.
|
| 40 |
+
- **8-bit, group size 64**: everything else: attention (including the MoVA value experts), the dense MLPs of the first layers, the embeddings and `lm_head`.
|
| 41 |
+
- **bf16**: the MoE and MoVA routers.
|
| 42 |
+
|
| 43 |
+
Plain 4-bit (everything 4-bit, group size 64) has a higher perplexity than this recipe: **+4.2%** WikiText-2 and **+2.1%** Arabic Wikipedia.
|
| 44 |
+
|
| 45 |
+
## Memory
|
| 46 |
+
|
| 47 |
+
Peak **26.6 GB** for a short prompt. The KV cache grows with the context, so long contexts need more memory; `--max-kv-size` and KV-cache quantization (`--kv-bits 8`, where available) reduce it.
|
| 48 |
+
|
| 49 |
+
## Conversion check
|
| 50 |
+
|
| 51 |
+
Checked against the PyTorch code (`modeling_k2_horizon.py`) in **fp32 on the real weights of this model** (rounded to bf16 on both sides, since the original checkpoint is fp32). The full model in fp32 needs more memory than the test machine has, so the check used the model truncated to its first 5 layers (the 3 dense layers and the first 2 MoE + MoVA layers, plus the final norm and `lm_head`), with the same truncation on both sides, on English, code, Arabic and math prompts: the logits match to a relative error of 6e-6 or better, the next-token predictions agree at every position, and greedy decoding with the KV cache gives the same tokens (64/64 on every prompt). The remaining MoE + MoVA layers run the same code path.
|
| 52 |
+
|
| 53 |
+
Smoke-tested after conversion with released mlx-lm 0.32.0 on English and Arabic chat prompts: answers end normally (the chat template and stop tokens work).
|
| 54 |
+
|
| 55 |
+
## Benchmarks (all Jais-3-MoVA-36B-A4B MLX variants)
|
| 56 |
+
|
| 57 |
+
Perplexity on 128 × 512 tokens (WikiText-2 test, and the first articles of the Arabic Wikipedia dump `20231101.ar`; lower is better) and generation speed on an Apple M4 Max, all measured the same way:
|
| 58 |
+
|
| 59 |
+
| | [bf16](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B-MLX-bf16) | [8-bit](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B-MLX-8bit) | [4-bit](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B-MLX-4bit) |
|
| 60 |
+
|---|---|---|---|
|
| 61 |
+
| Bits/weight | 16.0 | 8.5 | 5.6 |
|
| 62 |
+
| Disk | 74.9 GB | 39.8 GB | 26.4 GB |
|
| 63 |
+
| Peak memory | n/a | n/a | 26.6 GB |
|
| 64 |
+
| WikiText-2 perplexity (English) | n/a | n/a | 10.585 |
|
| 65 |
+
| Arabic Wikipedia perplexity | n/a | n/a | 13.143 |
|
| 66 |
+
| Generation | n/a | n/a | 65.2 tok/s |
|
| 67 |
+
|
| 68 |
+
The bf16 and 8-bit versions need more GPU memory than the 48 GB test machine allows by default, so they were not measured.
|
| 69 |
+
|
| 70 |
+
Perplexity is a coarse signal. Test the versions on your own workload before picking one.
|
| 71 |
+
|
| 72 |
+
## Usage
|
| 73 |
+
|
| 74 |
+
```bash
|
| 75 |
+
mlx_lm.generate --model IFM/Jais-3-MoVA-36B-A4B-MLX-4bit --trust-remote-code --prompt "ما هي عاصمة الإمارات؟" --max-tokens 2048
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
```python
|
| 79 |
+
from mlx_lm import load, generate
|
| 80 |
+
|
| 81 |
+
model, tokenizer = load("IFM/Jais-3-MoVA-36B-A4B-MLX-4bit", trust_remote_code=True)
|
| 82 |
+
messages = [{"role": "user", "content": "Explain mixture-of-experts in two sentences."}]
|
| 83 |
+
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
|
| 84 |
+
print(generate(model, tokenizer, prompt, max_tokens=2048))
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
`mlx_lm.chat` and `mlx_lm.server` take the same `--trust-remote-code` flag.
|
| 88 |
+
|
| 89 |
+
The model thinks before it answers, inside `<ifm|think> … </ifm|think>`. Leave room for that in `max_tokens`. The effort level is set with `reasoning_effort` in the chat template: `"high"` (default), `"medium"` or `"low"`, e.g. `apply_chat_template(messages, add_generation_prompt=True, reasoning_effort="low")`.
|
| 90 |
+
|
| 91 |
+
**Notes:**
|
| 92 |
+
|
| 93 |
+
- **Server output:** mlx-lm doesn't recognize the `<ifm|think>` tags yet, so `mlx_lm.server` returns the thinking text inside `content` rather than in a separate `reasoning` field. The tool-call format isn't parsed yet either.
|
| 94 |
+
- **Chat template change:** the original template raises an error when an earlier assistant message has no thinking field, which is what OpenAI-style clients send. The template here renders empty thinking for those messages instead. Nothing else was changed.
|
| 95 |
+
|
| 96 |
+
## License
|
| 97 |
+
|
| 98 |
+
[Apache-2.0](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B), inherited from the base model. Refer to the [original model card](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B) for architecture, benchmarks and intended use.
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,994 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{{- bos_token }}
|
| 2 |
+
{%- if tool_presentation is defined -%}
|
| 3 |
+
{{- raise_exception("Unsupported argument: tool_presentation. Use tool_presentation_format with one of: json, xml, markdown.") -}}
|
| 4 |
+
{%- endif -%}
|
| 5 |
+
{%- if tool_calling_format is defined -%}
|
| 6 |
+
{{- raise_exception("Unsupported argument: tool_calling_format. Use tool_call_format with one of: json, xml, xml_typed.") -}}
|
| 7 |
+
{%- endif -%}
|
| 8 |
+
{%- if tool_format is defined -%}
|
| 9 |
+
{{- raise_exception("Unsupported argument: tool_format. Use tool_call_format with one of: json, xml, xml_typed.") -}}
|
| 10 |
+
{%- endif -%}
|
| 11 |
+
{%- set tool_presentation_fmt = tool_presentation_format | default('markdown') -%}
|
| 12 |
+
{%- set tool_call_fmt = tool_call_format | default('xml') -%}
|
| 13 |
+
{%- if tool_presentation_fmt != 'json' and tool_presentation_fmt != 'xml' and tool_presentation_fmt != 'markdown' -%}
|
| 14 |
+
{{- raise_exception("Unsupported tool_presentation_format: '" ~ tool_presentation_fmt ~ "'. Supported formats: json, xml, markdown.") -}}
|
| 15 |
+
{%- endif -%}
|
| 16 |
+
{%- if tool_call_fmt != 'json' and tool_call_fmt != 'xml' and tool_call_fmt != 'xml_typed' -%}
|
| 17 |
+
{{- raise_exception("Unsupported tool_call_format: '" ~ tool_call_fmt ~ "'. Supported formats: json, xml, xml_typed.") -}}
|
| 18 |
+
{%- endif -%}
|
| 19 |
+
|
| 20 |
+
{#- Renderability state, computed during validate_tools (single walk, no extra -#}
|
| 21 |
+
{#- traversal at render time): ok = working flag for the tool being validated; -#}
|
| 22 |
+
{#- bad = pipe-delimited indices of tools that must render as verbatim JSON. -#}
|
| 23 |
+
{%- set RB = namespace(ok=true, bad='|') -%}
|
| 24 |
+
|
| 25 |
+
{%- macro value_contains_mapping(v) -%}
|
| 26 |
+
{%- if v is mapping -%}
|
| 27 |
+
true
|
| 28 |
+
{%- elif v is sequence and v is not string -%}
|
| 29 |
+
{%- set f = namespace(x='false') -%}
|
| 30 |
+
{%- for c in v -%}{%- if value_contains_mapping(c) == 'true' -%}{%- set f.x = 'true' -%}{%- endif -%}{%- endfor -%}
|
| 31 |
+
{{- f.x -}}
|
| 32 |
+
{%- else -%}
|
| 33 |
+
false
|
| 34 |
+
{%- endif -%}
|
| 35 |
+
{%- endmacro -%}
|
| 36 |
+
|
| 37 |
+
{#- $ref inlining state: defs = local $defs of the tool being rendered; seen = -#}
|
| 38 |
+
{#- pipe-delimited names already expanded for this tool (each def inlines at most -#}
|
| 39 |
+
{#- once; later references render by def name; cycles terminate immediately). -#}
|
| 40 |
+
{#- $ref-sibling annotations (description/default/...) merge OVER the def at -#}
|
| 41 |
+
{#- the inline site, so use-site annotations win and are never dropped. -#}
|
| 42 |
+
{%- set REFS = namespace(defs={}, seen='|') -%}
|
| 43 |
+
|
| 44 |
+
{%- macro render_compact_type_name(type_name, spec) -%}
|
| 45 |
+
{%- if type_name == "array" -%}
|
| 46 |
+
array[{%- if 'items' in spec -%}{{ render_compact_type(spec['items']) }}{%- else -%}any{%- endif -%}]
|
| 47 |
+
{%- elif type_name -%}
|
| 48 |
+
{{- type_name -}}
|
| 49 |
+
{%- else -%}
|
| 50 |
+
any
|
| 51 |
+
{%- endif -%}
|
| 52 |
+
{%- endmacro -%}
|
| 53 |
+
|
| 54 |
+
{%- macro render_compact_type(spec) -%}
|
| 55 |
+
{%- if spec is not mapping -%}
|
| 56 |
+
any
|
| 57 |
+
{%- elif spec.type is defined and spec.type is sequence and spec.type is not string and spec.type | length > 0 -%}
|
| 58 |
+
{%- for type_name in spec.type -%}{{ render_compact_type_name(type_name, spec) }}{%- if not loop.last -%}|{%- endif -%}{%- endfor -%}
|
| 59 |
+
{%- elif spec.type is defined and spec.type is sequence and spec.type is not string -%}
|
| 60 |
+
any
|
| 61 |
+
{%- elif spec.type -%}
|
| 62 |
+
{{- render_compact_type_name(spec.type, spec) -}}
|
| 63 |
+
{%- elif spec['$ref'] is string -%}
|
| 64 |
+
{{- spec['$ref'].split('/') | last -}}
|
| 65 |
+
{%- elif spec.oneOf -%}
|
| 66 |
+
oneOf[{%- for variant in spec.oneOf -%}{{ render_compact_type(variant) }}{%- if not loop.last -%}|{%- endif -%}{%- endfor -%}]
|
| 67 |
+
{%- elif spec.anyOf -%}
|
| 68 |
+
anyOf[{%- for variant in spec.anyOf -%}{{ render_compact_type(variant) }}{%- if not loop.last -%}|{%- endif -%}{%- endfor -%}]
|
| 69 |
+
{%- elif spec.properties -%}
|
| 70 |
+
object
|
| 71 |
+
{%- elif 'items' in spec -%}
|
| 72 |
+
array[{{ render_compact_type(spec['items']) }}]
|
| 73 |
+
{%- else -%}
|
| 74 |
+
any
|
| 75 |
+
{%- endif -%}
|
| 76 |
+
{%- endmacro -%}
|
| 77 |
+
|
| 78 |
+
{%- macro render_markdown_type_name(type_name, spec) -%}
|
| 79 |
+
{%- if type_name == "array" -%}
|
| 80 |
+
array of {% if 'items' in spec %}{{ render_markdown_type(spec['items']) }}{% else %}any{% endif %}
|
| 81 |
+
{%- elif type_name -%}
|
| 82 |
+
{{- type_name -}}
|
| 83 |
+
{%- else -%}
|
| 84 |
+
any
|
| 85 |
+
{%- endif -%}
|
| 86 |
+
{%- endmacro -%}
|
| 87 |
+
|
| 88 |
+
{%- macro render_markdown_type(spec) -%}
|
| 89 |
+
{%- if spec is sameas true -%}
|
| 90 |
+
True
|
| 91 |
+
{%- elif spec is sameas false -%}
|
| 92 |
+
False
|
| 93 |
+
{%- elif spec is not mapping -%}
|
| 94 |
+
any
|
| 95 |
+
{%- elif spec.type is defined and spec.type is sequence and spec.type is not string and spec.type | length > 0 -%}
|
| 96 |
+
{%- for type_name in spec.type -%}{{ render_markdown_type_name(type_name, spec) }}{% if not loop.last %} or {% endif %}{%- endfor -%}
|
| 97 |
+
{%- elif spec.type is defined and spec.type is sequence and spec.type is not string -%}
|
| 98 |
+
any
|
| 99 |
+
{%- elif spec.type -%}
|
| 100 |
+
{{- render_markdown_type_name(spec.type, spec) -}}
|
| 101 |
+
{%- elif spec['$ref'] is string -%}
|
| 102 |
+
{{- spec['$ref'].split('/') | last -}}
|
| 103 |
+
{%- elif spec.oneOf -%}
|
| 104 |
+
oneOf[{%- for variant in spec.oneOf -%}{{ render_markdown_type(variant) }}{% if not loop.last %} or {% endif %}{%- endfor -%}]
|
| 105 |
+
{%- elif spec.anyOf -%}
|
| 106 |
+
anyOf[{%- for variant in spec.anyOf -%}{{ render_markdown_type(variant) }}{% if not loop.last %} or {% endif %}{%- endfor -%}]
|
| 107 |
+
{%- elif spec.properties -%}
|
| 108 |
+
object
|
| 109 |
+
{%- elif 'items' in spec -%}
|
| 110 |
+
array of {{ render_markdown_type(spec['items']) }}
|
| 111 |
+
{%- else -%}
|
| 112 |
+
any
|
| 113 |
+
{%- endif -%}
|
| 114 |
+
{%- endmacro -%}
|
| 115 |
+
|
| 116 |
+
{%- macro render_xml_text(value) -%}
|
| 117 |
+
{{- value.split() | join(" ") -}}
|
| 118 |
+
{%- endmacro -%}
|
| 119 |
+
|
| 120 |
+
{%- macro render_python_string(value) -%}
|
| 121 |
+
'{{- value.split() | join(" ") | replace("\\", "\\\\") | replace("'", "\\'") -}}'
|
| 122 |
+
{%- endmacro -%}
|
| 123 |
+
|
| 124 |
+
{%- macro render_python_repr(value) -%}
|
| 125 |
+
{%- if value is string -%}
|
| 126 |
+
{{ render_python_string(value) }}
|
| 127 |
+
{%- elif value is sameas true -%}
|
| 128 |
+
True
|
| 129 |
+
{%- elif value is sameas false -%}
|
| 130 |
+
False
|
| 131 |
+
{%- elif value is none -%}
|
| 132 |
+
None
|
| 133 |
+
{%- elif value is mapping -%}
|
| 134 |
+
{{- "{" -}}
|
| 135 |
+
{%- for key, child in value | items -%}
|
| 136 |
+
{{ render_python_repr(key) }}: {{ render_python_repr(child) }}{%- if not loop.last -%}, {% endif -%}
|
| 137 |
+
{%- endfor -%}
|
| 138 |
+
{{- "}" -}}
|
| 139 |
+
{%- elif value is sequence -%}
|
| 140 |
+
{{- "[" -}}
|
| 141 |
+
{%- for child in value -%}
|
| 142 |
+
{{ render_python_repr(child) }}{%- if not loop.last -%}, {% endif -%}
|
| 143 |
+
{%- endfor -%}
|
| 144 |
+
{{- "]" -}}
|
| 145 |
+
{%- else -%}
|
| 146 |
+
{{- value -}}
|
| 147 |
+
{%- endif -%}
|
| 148 |
+
{%- endmacro -%}
|
| 149 |
+
|
| 150 |
+
{%- macro render_xml_value(value) -%}
|
| 151 |
+
{%- if value is string -%}{{ render_xml_text(value) }}{%- else -%}{{ render_python_repr(value) }}{%- endif -%}
|
| 152 |
+
{%- endmacro -%}
|
| 153 |
+
|
| 154 |
+
{%- macro render_xml_enum_value(value) -%}
|
| 155 |
+
{%- if value is string -%}"{{- value | replace("\\", "\\\\") | replace("\"", "\\\"") -}}"{%- else -%}"{{- render_python_repr(value) | replace("\\", "\\\\") | replace("\"", "\\\"") -}}"{%- endif -%}
|
| 156 |
+
{%- endmacro -%}
|
| 157 |
+
|
| 158 |
+
{%- macro render_xml_enum(values) -%}
|
| 159 |
+
{%- for value in values -%}{{ render_xml_enum_value(value) }}{%- if not loop.last -%}|{%- endif -%}{%- endfor -%}
|
| 160 |
+
{%- endmacro -%}
|
| 161 |
+
|
| 162 |
+
{%- macro render_xml_default_attr(value) -%}
|
| 163 |
+
{{- " default=" }}{%- if value is string -%}"{{- value | replace("\\", "\\\\") | replace("\"", "\\\"") -}}"{%- else -%}{{ render_xml_value(value) }}{%- endif -%}
|
| 164 |
+
{%- endmacro -%}
|
| 165 |
+
|
| 166 |
+
{%- macro render_xml_attr(name, value) -%}
|
| 167 |
+
{{- " " + name + "=" }}{%- if value == "" -%}""{%- else -%}{{ render_xml_value(value) }}{%- endif -%}
|
| 168 |
+
{%- endmacro -%}
|
| 169 |
+
|
| 170 |
+
{%- macro validate_schema(spec, path, lenient=false, classify=true, in_variant=false) -%}
|
| 171 |
+
{%- if spec is mapping -%}
|
| 172 |
+
{%- if not lenient -%}
|
| 173 |
+
{%- if spec.required is defined -%}
|
| 174 |
+
{%- if spec.required is string or spec.required is not sequence -%}
|
| 175 |
+
{{- raise_exception("Schema '" + path + "' has 'required' but it is not a list.") -}}
|
| 176 |
+
{%- endif -%}
|
| 177 |
+
{%- if spec.required | length > 0 and not spec.properties and not in_variant -%}
|
| 178 |
+
{{- raise_exception("Schema '" + path + "' has required fields but no properties object to define them.") -}}
|
| 179 |
+
{%- endif -%}
|
| 180 |
+
{%- if spec.properties -%}
|
| 181 |
+
{%- for required_name in spec.required -%}
|
| 182 |
+
{%- if required_name not in spec.properties -%}
|
| 183 |
+
{{- raise_exception("Schema '" + path + "' marks '" + required_name + "' as required, but that property is not defined in properties.") -}}
|
| 184 |
+
{%- endif -%}
|
| 185 |
+
{%- endfor -%}
|
| 186 |
+
{%- endif -%}
|
| 187 |
+
{%- endif -%}
|
| 188 |
+
{%- endif -%}
|
| 189 |
+
{#- renderability classification, piggybacking on this walk (no raises here): -#}
|
| 190 |
+
{#- constructs the pretty renderer does not fully handle flip RB.ok so the -#}
|
| 191 |
+
{#- tool falls back to verbatim JSON. Skipped entirely for json presentation. -#}
|
| 192 |
+
{%- if classify -%}
|
| 193 |
+
{%- for key, value in spec | items -%}
|
| 194 |
+
{%- if key == '$ref' -%}
|
| 195 |
+
{%- if value is not string -%}{%- set RB.ok = false -%}
|
| 196 |
+
{%- elif not (value.startswith('#/$defs/') or value.startswith('#/definitions/')) -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 197 |
+
{%- elif key == '$defs' or key == 'definitions' -%}
|
| 198 |
+
{%- if value is mapping -%}
|
| 199 |
+
{%- for dk, dv in value | items -%}
|
| 200 |
+
{{- validate_schema(dv, path + ".$defs." + dk, true) -}}
|
| 201 |
+
{%- endfor -%}
|
| 202 |
+
{%- else -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 203 |
+
{%- elif key == 'type' -%}
|
| 204 |
+
{%- if value is mapping -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 205 |
+
{%- elif key == 'enum' -%}
|
| 206 |
+
{%- if value is string or value is mapping or value is not sequence -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 207 |
+
{%- elif key == 'items' -%}
|
| 208 |
+
{#- any items shape renders: mapping structurally, others via repr detail -#}
|
| 209 |
+
{%- elif key == 'oneOf' or key == 'anyOf' -%}
|
| 210 |
+
{%- if value is mapping or value is string or value is not sequence -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 211 |
+
{%- elif key == 'required' -%}
|
| 212 |
+
{%- if value and not spec.properties -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 213 |
+
{%- elif ('|' ~ key ~ '|') in '|description|default|title|examples|properties|patternProperties|additionalProperties|returns|' -%}
|
| 214 |
+
{%- elif value is mapping -%}
|
| 215 |
+
{%- for uk, uv in value | items -%}
|
| 216 |
+
{%- if value_contains_mapping(uv) == 'true' -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 217 |
+
{%- endfor -%}
|
| 218 |
+
{%- elif value is sequence and value is not string -%}
|
| 219 |
+
{%- if value_contains_mapping(value) == 'true' -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 220 |
+
{%- endif -%}
|
| 221 |
+
{%- endfor -%}
|
| 222 |
+
{%- endif -%}
|
| 223 |
+
{%- if spec.properties -%}
|
| 224 |
+
{%- for child_name, child_spec in spec.properties | items -%}
|
| 225 |
+
{{- validate_schema(child_spec, path + "." + child_name, lenient, classify) -}}
|
| 226 |
+
{%- endfor -%}
|
| 227 |
+
{%- endif -%}
|
| 228 |
+
{%- if 'items' in spec -%}{{- validate_schema(spec['items'], path + "[]", lenient, classify) -}}{%- endif -%}
|
| 229 |
+
{%- if spec.oneOf -%}
|
| 230 |
+
{%- for variant in spec.oneOf -%}{{- validate_schema(variant, path + ".oneOf[" + (loop.index0 | string) + "]", lenient, classify, true) -}}{%- endfor -%}
|
| 231 |
+
{%- endif -%}
|
| 232 |
+
{%- if spec.anyOf -%}
|
| 233 |
+
{%- for variant in spec.anyOf -%}{{- validate_schema(variant, path + ".anyOf[" + (loop.index0 | string) + "]", lenient, classify, true) -}}{%- endfor -%}
|
| 234 |
+
{%- endif -%}
|
| 235 |
+
{%- if spec.additionalProperties is mapping -%}{{- validate_schema(spec.additionalProperties, path + ".additionalProperties", lenient, classify) -}}{%- endif -%}
|
| 236 |
+
{%- if spec.patternProperties is mapping -%}
|
| 237 |
+
{%- for pattern, pattern_spec in spec.patternProperties | items -%}
|
| 238 |
+
{{- validate_schema(pattern_spec, path + ".patternProperties[" + pattern + "]", lenient, classify) -}}
|
| 239 |
+
{%- endfor -%}
|
| 240 |
+
{%- endif -%}
|
| 241 |
+
{%- if spec.returns is mapping -%}{{- validate_schema(spec.returns, path + ".returns", lenient, classify) -}}{%- endif -%}
|
| 242 |
+
{%- endif -%}
|
| 243 |
+
{%- endmacro -%}
|
| 244 |
+
|
| 245 |
+
{%- macro validate_tools(tools_list, classify=true) -%}
|
| 246 |
+
{%- set RB.bad = '|' -%}
|
| 247 |
+
{%- for tool in tools_list -%}
|
| 248 |
+
{%- set fn = tool.function if tool.function is defined else tool -%}
|
| 249 |
+
{%- set RB.ok = true -%}
|
| 250 |
+
{%- if fn.parameters is defined and fn.parameters is string -%}
|
| 251 |
+
{{- raise_exception("tool.function.parameters must be a dict, not a JSON string. Parse it before passing to the template.") -}}
|
| 252 |
+
{%- endif -%}
|
| 253 |
+
{%- if fn.parameters is not defined or fn.parameters is none -%}
|
| 254 |
+
{%- if fn.arguments is defined -%}
|
| 255 |
+
{{- raise_exception("Tool '" + fn.name + "' has 'arguments' instead of 'parameters'. Rename 'arguments' to 'parameters'.") -}}
|
| 256 |
+
{%- else -%}
|
| 257 |
+
{{- raise_exception("Tool '" + fn.name + "' is missing required 'parameters' field. Each tool must have a 'parameters' dict with 'type', 'properties', and 'required' keys.") -}}
|
| 258 |
+
{%- endif -%}
|
| 259 |
+
{%- endif -%}
|
| 260 |
+
{{- validate_schema(fn.parameters, "tool." + fn.name + ".parameters", false, classify) -}}
|
| 261 |
+
{%- if classify -%}
|
| 262 |
+
{%- if fn.parameters is mapping -%}
|
| 263 |
+
{#- unknown container-valued keys at the parameters ROOT are never rendered -#}
|
| 264 |
+
{#- by the pretty path (root extras are dropped) -> verbatim fallback. -#}
|
| 265 |
+
{%- for rk, rv in fn.parameters | items -%}
|
| 266 |
+
{%- if rk not in ['type', 'description', 'enum', 'default', 'properties', 'required', 'optional', 'title', 'items', 'oneOf', 'anyOf', 'additionalProperties', 'patternProperties', 'returns', 'examples', '$defs', 'definitions', '$ref'] -%}
|
| 267 |
+
{%- if rv is mapping or (rv is sequence and rv is not string) -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 268 |
+
{%- endif -%}
|
| 269 |
+
{%- endfor -%}
|
| 270 |
+
{%- else -%}
|
| 271 |
+
{%- set RB.ok = false -%}
|
| 272 |
+
{%- endif -%}
|
| 273 |
+
{%- endif -%}
|
| 274 |
+
{%- if fn.returns is mapping -%}{{- validate_schema(fn.returns, "tool." + fn.name + ".returns", false, classify) -}}{%- endif -%}
|
| 275 |
+
{%- if classify and fn.returns is not defined and fn.response is mapping -%}{{- validate_schema(fn.response, "tool." + fn.name + ".response", true) -}}{%- endif -%}
|
| 276 |
+
{#- unknown container-valued keys at the FUNCTION level are never rendered -> fallback. -#}
|
| 277 |
+
{%- if classify -%}
|
| 278 |
+
{%- for fk, fv in fn | items -%}
|
| 279 |
+
{%- if fk not in ['name', 'description', 'parameters', 'returns', 'response', 'type', 'function'] -%}
|
| 280 |
+
{%- if fv is mapping or (fv is sequence and fv is not string) -%}{%- set RB.ok = false -%}{%- endif -%}
|
| 281 |
+
{%- endif -%}
|
| 282 |
+
{%- endfor -%}
|
| 283 |
+
{%- endif -%}
|
| 284 |
+
{%- if not RB.ok -%}{%- set RB.bad = RB.bad ~ loop.index0 ~ '|' -%}{%- endif -%}
|
| 285 |
+
{%- endfor -%}
|
| 286 |
+
{%- endmacro -%}
|
| 287 |
+
|
| 288 |
+
{%- macro render_tools_json(tools_list) -%}
|
| 289 |
+
{{- "<ifm|tools>" }}
|
| 290 |
+
{%- for tool in tools_list %}
|
| 291 |
+
{{- "\n" }}
|
| 292 |
+
{{- tool | tojson }}
|
| 293 |
+
{%- endfor %}
|
| 294 |
+
{{- "\n</ifm|tools>" }}
|
| 295 |
+
{%- endmacro -%}
|
| 296 |
+
|
| 297 |
+
{%- macro render_xml_schema_attrs(spec, include_value_attrs) -%}
|
| 298 |
+
{%- if spec is mapping -%}
|
| 299 |
+
{%- set structural_keys = ["type", "description", "enum", "default", "properties", "required", "items", "oneOf", "anyOf", "additionalProperties", "patternProperties", "returns"] -%}
|
| 300 |
+
{%- if include_value_attrs and spec.enum -%}{{- " enum=" }}{{ render_xml_enum(spec.enum) }}{%- endif -%}
|
| 301 |
+
{%- if include_value_attrs and spec.default is defined -%}{{ render_xml_default_attr(spec.default) }}{%- endif -%}
|
| 302 |
+
{%- if spec.additionalProperties is defined and spec.additionalProperties is not mapping -%}{{ render_xml_attr("additionalProperties", spec.additionalProperties) }}{%- endif -%}
|
| 303 |
+
{%- if spec.patternProperties is defined and spec.patternProperties is not mapping -%}{{ render_xml_attr("patternProperties", spec.patternProperties) }}{%- endif -%}
|
| 304 |
+
{%- for key, value in spec | items -%}
|
| 305 |
+
{%- if key not in structural_keys -%}
|
| 306 |
+
{{ render_xml_attr(key, value) }}
|
| 307 |
+
{%- endif -%}
|
| 308 |
+
{%- endfor -%}
|
| 309 |
+
{%- endif -%}
|
| 310 |
+
{%- endmacro -%}
|
| 311 |
+
|
| 312 |
+
{%- macro xml_schema_has_children(spec, include_properties, include_description) -%}
|
| 313 |
+
{%- if spec is not mapping -%}
|
| 314 |
+
false
|
| 315 |
+
{%- elif (include_description and spec.description is defined) or (include_properties and spec.properties) or 'items' in spec or spec.oneOf or spec.anyOf or spec.additionalProperties is mapping or spec.patternProperties is mapping or spec.returns is defined -%}
|
| 316 |
+
true
|
| 317 |
+
{%- else -%}
|
| 318 |
+
false
|
| 319 |
+
{%- endif -%}
|
| 320 |
+
{%- endmacro -%}
|
| 321 |
+
|
| 322 |
+
{%- macro render_xml_schema_node(tag, spec, include_properties) -%}
|
| 323 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 324 |
+
{%- set _r = spec['$ref'] -%}
|
| 325 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 326 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 327 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 328 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 329 |
+
{%- if spec['$ref'] is string -%}
|
| 330 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 331 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 332 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 333 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 334 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 335 |
+
{%- endif -%}
|
| 336 |
+
{%- endif -%}
|
| 337 |
+
{%- endif -%}
|
| 338 |
+
{%- endif -%}
|
| 339 |
+
{%- if spec is mapping -%}
|
| 340 |
+
{{- "<" + tag + " type=" + render_compact_type(spec) }}{{ render_xml_schema_attrs(spec, true) }}
|
| 341 |
+
{%- if xml_schema_has_children(spec, include_properties, true) == 'true' -%}
|
| 342 |
+
{{- ">" }}{{ render_xml_schema_children(spec, include_properties, true) }}{{- "</" + tag + ">" }}
|
| 343 |
+
{%- else -%}
|
| 344 |
+
{{- "/>" }}
|
| 345 |
+
{%- endif -%}
|
| 346 |
+
{%- else -%}
|
| 347 |
+
{{- "<" + tag + ">" }}{{ render_xml_value(spec) }}{{- "</" + tag + ">" }}
|
| 348 |
+
{%- endif -%}
|
| 349 |
+
{%- endmacro -%}
|
| 350 |
+
|
| 351 |
+
{%- macro render_xml_pattern_property(pattern, spec) -%}
|
| 352 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 353 |
+
{%- set _r = spec['$ref'] -%}
|
| 354 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 355 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 356 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 357 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 358 |
+
{%- if spec['$ref'] is string -%}
|
| 359 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 360 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 361 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 362 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 363 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 364 |
+
{%- endif -%}
|
| 365 |
+
{%- endif -%}
|
| 366 |
+
{%- endif -%}
|
| 367 |
+
{%- endif -%}
|
| 368 |
+
{%- if spec is mapping -%}
|
| 369 |
+
{{- "<patternProperty" }}{{ render_xml_attr("pattern", pattern) }}{{- " type=" + render_compact_type(spec) }}{{ render_xml_schema_attrs(spec, true) }}
|
| 370 |
+
{%- if xml_schema_has_children(spec, true, true) == 'true' -%}
|
| 371 |
+
{{- ">" }}{{ render_xml_schema_children(spec, true, true) }}{{- "</patternProperty>" }}
|
| 372 |
+
{%- else -%}
|
| 373 |
+
{{- "/>" }}
|
| 374 |
+
{%- endif -%}
|
| 375 |
+
{%- else -%}
|
| 376 |
+
{{- "<patternProperty" }}{{ render_xml_attr("pattern", pattern) }}{{- ">" }}{{ render_xml_value(spec) }}{{- "</patternProperty>" }}
|
| 377 |
+
{%- endif -%}
|
| 378 |
+
{%- endmacro -%}
|
| 379 |
+
|
| 380 |
+
{%- macro render_xml_schema_children(spec, include_properties, include_description) -%}
|
| 381 |
+
{%- if include_description and spec.description is defined -%}{{- "<description>" }}{{ spec.description }}{{- "</description>" }}{%- endif -%}
|
| 382 |
+
{%- if include_properties and spec.properties -%}
|
| 383 |
+
{%- for child_name, child_spec in spec.properties | items -%}
|
| 384 |
+
{{- render_xml_param(child_name, child_spec, spec.required or []) }}
|
| 385 |
+
{%- endfor -%}
|
| 386 |
+
{%- endif -%}
|
| 387 |
+
{%- if 'items' in spec -%}{{ render_xml_schema_node("items", spec['items'], true) }}{%- endif -%}
|
| 388 |
+
{%- if spec.oneOf -%}
|
| 389 |
+
{{- "<oneOf>" }}
|
| 390 |
+
{%- for variant in spec.oneOf -%}{{ render_xml_schema_node("variant", variant, true) }}{%- endfor -%}
|
| 391 |
+
{{- "</oneOf>" }}
|
| 392 |
+
{%- endif -%}
|
| 393 |
+
{%- if spec.anyOf -%}
|
| 394 |
+
{{- "<anyOf>" }}
|
| 395 |
+
{%- for variant in spec.anyOf -%}{{ render_xml_schema_node("variant", variant, true) }}{%- endfor -%}
|
| 396 |
+
{{- "</anyOf>" }}
|
| 397 |
+
{%- endif -%}
|
| 398 |
+
{%- if spec.additionalProperties is mapping -%}{{ render_xml_schema_node("additionalProperties", spec.additionalProperties, true) }}{%- endif -%}
|
| 399 |
+
{%- if spec.patternProperties is mapping -%}
|
| 400 |
+
{{- "<patternProperties>" }}
|
| 401 |
+
{%- for pattern, pattern_spec in spec.patternProperties | items -%}{{ render_xml_pattern_property(pattern, pattern_spec) }}{%- endfor -%}
|
| 402 |
+
{{- "</patternProperties>" }}
|
| 403 |
+
{%- elif spec.patternProperties is defined -%}<patternProperties>{{ render_xml_value(spec.patternProperties) }}</patternProperties>{%- endif -%}
|
| 404 |
+
{%- if spec.returns is mapping -%}{{ render_xml_schema_node("returns", spec.returns, true) }}{%- elif spec.returns is defined -%}<returns>{{ render_xml_value(spec.returns) }}</returns>{%- endif -%}
|
| 405 |
+
{%- endmacro -%}
|
| 406 |
+
|
| 407 |
+
{%- macro render_xml_param(name, spec, required_list) -%}
|
| 408 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 409 |
+
{%- set _r = spec['$ref'] -%}
|
| 410 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 411 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 412 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 413 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 414 |
+
{%- if spec['$ref'] is string -%}
|
| 415 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 416 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 417 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 418 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 419 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 420 |
+
{%- endif -%}
|
| 421 |
+
{%- endif -%}
|
| 422 |
+
{%- endif -%}
|
| 423 |
+
{%- endif -%}
|
| 424 |
+
{{- "<param name=" + name + " type=" + render_compact_type(spec) }}
|
| 425 |
+
{%- if name in (required_list or []) -%}{{- " required=true" }}{%- endif -%}
|
| 426 |
+
{%- if spec.enum -%}{{- " enum=" }}{{ render_xml_enum(spec.enum) }}{%- endif -%}
|
| 427 |
+
{%- if spec.default is defined -%}{{ render_xml_default_attr(spec.default) }}{%- endif -%}
|
| 428 |
+
{{- render_xml_schema_attrs(spec, false) }}
|
| 429 |
+
{%- if spec.description or xml_schema_has_children(spec, true, false) == 'true' -%}
|
| 430 |
+
{{- ">" }}
|
| 431 |
+
{%- if spec.description -%}{{ spec.description }}{%- endif -%}
|
| 432 |
+
{{- render_xml_schema_children(spec, true, false) }}
|
| 433 |
+
{{- "</param>" }}
|
| 434 |
+
{%- else -%}
|
| 435 |
+
{{- "/>" }}
|
| 436 |
+
{%- endif -%}
|
| 437 |
+
{%- endmacro -%}
|
| 438 |
+
|
| 439 |
+
{%- macro render_tools_xml(tools_list) -%}
|
| 440 |
+
{{- "<ifm|tools>" }}
|
| 441 |
+
{%- for tool in tools_list -%}
|
| 442 |
+
{%- set fn = tool.function if tool.function is defined else tool -%}
|
| 443 |
+
{%- set REFS.defs = fn.parameters['$defs'] if (fn.parameters is mapping and fn.parameters['$defs'] is mapping) else (fn.parameters['definitions'] if (fn.parameters is mapping and fn.parameters['definitions'] is mapping) else {}) -%}
|
| 444 |
+
{%- set REFS.seen = '|' -%}
|
| 445 |
+
{%- set fnp = namespace(p=fn.parameters) -%}
|
| 446 |
+
{%- if fnp.p is mapping and fnp.p['$ref'] is string -%}
|
| 447 |
+
{%- set _r = fnp.p['$ref'] -%}
|
| 448 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 449 |
+
{%- if _k is not none and REFS.defs[_k] is mapping -%}
|
| 450 |
+
{%- set fnp.p = dict((REFS.defs[_k] | items | list) + (fnp.p | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 451 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 452 |
+
{%- endif -%}
|
| 453 |
+
{%- endif -%}
|
| 454 |
+
{{- "\n<function name=" + fn.name + ">" }}
|
| 455 |
+
{%- if fn.description -%}
|
| 456 |
+
{{- "<description>" }}{{ fn.description }}{{- "</description>" }}
|
| 457 |
+
{%- endif -%}
|
| 458 |
+
{{- "<parameters>" }}
|
| 459 |
+
{%- if fnp.p and fnp.p.properties -%}
|
| 460 |
+
{%- for pname, pspec in fnp.p.properties | items -%}
|
| 461 |
+
{{- render_xml_param(pname, pspec, fnp.p.required or []) }}
|
| 462 |
+
{%- endfor -%}
|
| 463 |
+
{%- elif fnp.p is mapping and (fnp.p.oneOf or fnp.p.anyOf or 'items' in fnp.p) -%}
|
| 464 |
+
{{- render_xml_schema_children(fnp.p, true, false) }}
|
| 465 |
+
{%- endif -%}
|
| 466 |
+
{{- "</parameters>" }}
|
| 467 |
+
{%- set fn_ret = fn.returns if fn.returns is defined else fn.response -%}
|
| 468 |
+
{%- if fn_ret is mapping -%}{{ render_xml_schema_node("returns", fn_ret, true) }}{%- elif fn_ret is defined -%}<returns>{{ render_xml_value(fn_ret) }}</returns>{%- endif -%}
|
| 469 |
+
{{- "</function>" }}
|
| 470 |
+
{%- endfor -%}
|
| 471 |
+
{{- "\n</ifm|tools>" }}
|
| 472 |
+
{%- endmacro -%}
|
| 473 |
+
|
| 474 |
+
{%- macro render_markdown_literal(value) -%}
|
| 475 |
+
{%- if value is string and value == "" -%}""
|
| 476 |
+
{%- elif value is string -%}`{{ value | replace("\n", "\\n") }}`
|
| 477 |
+
{%- else -%}`{{ render_python_repr(value) }}`
|
| 478 |
+
{%- endif -%}
|
| 479 |
+
{%- endmacro -%}
|
| 480 |
+
|
| 481 |
+
{%- macro render_allowed_values(values) -%}
|
| 482 |
+
{%- for value in values -%}{{ render_markdown_literal(value) }}{% if not loop.last %}, {% endif %}{%- endfor -%}
|
| 483 |
+
{%- endmacro -%}
|
| 484 |
+
|
| 485 |
+
{%- macro render_markdown_value(value) -%}
|
| 486 |
+
{%- if value is string and value == "" -%}""{%- elif value is string -%}{{ value }}{%- else -%}{{ render_python_repr(value) }}{%- endif -%}
|
| 487 |
+
{%- endmacro -%}
|
| 488 |
+
|
| 489 |
+
{%- macro render_markdown_detail(indent, label, value) -%}
|
| 490 |
+
{{- "\n" + indent + " - " + label + ": " }}{{ render_markdown_value(value) }}
|
| 491 |
+
{%- endmacro -%}
|
| 492 |
+
|
| 493 |
+
{%- macro render_markdown_metadata_detail(label, value) -%}
|
| 494 |
+
{{- "\n- " + label + ": " }}{{ render_markdown_value(value) }}
|
| 495 |
+
{%- endmacro -%}
|
| 496 |
+
|
| 497 |
+
{%- macro render_markdown_schema_annotations(spec, indent, include_value_details) -%}
|
| 498 |
+
{%- if include_value_details and spec.description is defined -%}{{ render_markdown_detail(indent, "Description", spec.description | replace("\n", "\n" + indent + " ")) }}{%- endif -%}
|
| 499 |
+
{%- if include_value_details and spec.enum is defined -%}{{- "\n" + indent + " - Allowed values: " }}{{ render_allowed_values(spec.enum) }}{%- endif -%}
|
| 500 |
+
{%- if include_value_details and spec.default is defined -%}{{- "\n" + indent + " - Default: " }}{{ render_markdown_literal(spec.default) }}{%- endif -%}
|
| 501 |
+
{%- if spec.additionalProperties is defined -%}
|
| 502 |
+
{%- if spec.additionalProperties is mapping -%}
|
| 503 |
+
{{- "\n" + indent + " - Additional properties *(" + render_markdown_type(spec.additionalProperties) + ")*" }}
|
| 504 |
+
{{- render_markdown_schema_details(spec.additionalProperties, indent + " ", true) }}
|
| 505 |
+
{%- else -%}
|
| 506 |
+
{{ render_markdown_detail(indent, "Additional properties", spec.additionalProperties) }}
|
| 507 |
+
{%- endif -%}
|
| 508 |
+
{%- endif -%}
|
| 509 |
+
{%- endmacro -%}
|
| 510 |
+
|
| 511 |
+
{%- macro render_markdown_metadata_annotations(spec) -%}
|
| 512 |
+
{%- if spec.description is defined -%}{{ render_markdown_metadata_detail("Description", spec.description | replace("\n", "\n ")) }}{%- endif -%}
|
| 513 |
+
{%- if spec.enum is defined -%}{{- "\n- Allowed values: " }}{{ render_allowed_values(spec.enum) }}{%- endif -%}
|
| 514 |
+
{%- if spec.default is defined -%}{{- "\n- Default: " }}{{ render_markdown_literal(spec.default) }}{%- endif -%}
|
| 515 |
+
{%- if spec.additionalProperties is defined -%}
|
| 516 |
+
{%- if spec.additionalProperties is mapping -%}
|
| 517 |
+
{{- "\n- Additional properties *(" + render_markdown_type(spec.additionalProperties) + ")*" }}
|
| 518 |
+
{{- render_markdown_schema_details(spec.additionalProperties, "", true) }}
|
| 519 |
+
{%- else -%}
|
| 520 |
+
{{ render_markdown_metadata_detail("Additional properties", spec.additionalProperties) }}
|
| 521 |
+
{%- endif -%}
|
| 522 |
+
{%- endif -%}
|
| 523 |
+
{%- endmacro -%}
|
| 524 |
+
|
| 525 |
+
{%- macro render_markdown_schema_extras(spec, indent) -%}
|
| 526 |
+
{%- set rendered_keys = ["type", "description", "enum", "default", "properties", "required", "items", "oneOf", "anyOf", "additionalProperties", "patternProperties", "returns"] -%}
|
| 527 |
+
{%- for key, value in spec | items -%}
|
| 528 |
+
{%- if key not in rendered_keys -%}
|
| 529 |
+
{{- "\n" + indent + " - " + key + ": " }}{{ render_markdown_value(value) }}
|
| 530 |
+
{%- endif -%}
|
| 531 |
+
{%- endfor -%}
|
| 532 |
+
{%- endmacro -%}
|
| 533 |
+
|
| 534 |
+
{%- macro render_markdown_metadata_extras(spec) -%}
|
| 535 |
+
{%- set rendered_keys = ["type", "description", "enum", "default", "properties", "required", "items", "oneOf", "anyOf", "additionalProperties", "patternProperties", "returns"] -%}
|
| 536 |
+
{%- for key, value in spec | items -%}
|
| 537 |
+
{%- if key not in rendered_keys -%}
|
| 538 |
+
{{- "\n- " + key + ": " }}{{ render_markdown_value(value) }}
|
| 539 |
+
{%- endif -%}
|
| 540 |
+
{%- endfor -%}
|
| 541 |
+
{%- endmacro -%}
|
| 542 |
+
|
| 543 |
+
{%- macro markdown_schema_has_extra(spec) -%}
|
| 544 |
+
{%- set rendered_keys = ["type", "description", "enum", "default", "properties", "required", "items", "oneOf", "anyOf", "additionalProperties", "patternProperties", "returns"] -%}
|
| 545 |
+
{%- set found = namespace(value='false') -%}
|
| 546 |
+
{%- for key, value in spec | items -%}
|
| 547 |
+
{%- if key not in rendered_keys -%}{%- set found.value = 'true' -%}{%- endif -%}
|
| 548 |
+
{%- endfor -%}
|
| 549 |
+
{{- found.value -}}
|
| 550 |
+
{%- endmacro -%}
|
| 551 |
+
|
| 552 |
+
{%- macro markdown_parameter_schema_has_details(spec) -%}
|
| 553 |
+
{%- if spec.description is defined or spec.enum is defined or spec.default is defined or spec.additionalProperties is defined or spec.patternProperties is defined or 'items' in spec or spec.oneOf or spec.anyOf or spec.returns is defined or markdown_schema_has_extra(spec) == 'true' -%}
|
| 554 |
+
true
|
| 555 |
+
{%- else -%}
|
| 556 |
+
false
|
| 557 |
+
{%- endif -%}
|
| 558 |
+
{%- endmacro -%}
|
| 559 |
+
|
| 560 |
+
{%- macro render_markdown_schema_structure(spec, indent, include_properties) -%}
|
| 561 |
+
{%- if include_properties and spec.properties -%}
|
| 562 |
+
{%- for child_name, child_spec in spec.properties | items -%}
|
| 563 |
+
{{- render_markdown_param(child_name, child_spec, spec.required or [], indent + " ") }}
|
| 564 |
+
{%- endfor -%}
|
| 565 |
+
{%- endif -%}
|
| 566 |
+
{%- if 'items' in spec and spec['items'] is mapping -%}
|
| 567 |
+
{{- "\n" + indent + " - Items *(" + render_markdown_type(spec['items']) + ")*" }}
|
| 568 |
+
{{- render_markdown_schema_details(spec['items'], indent + " ", true) }}
|
| 569 |
+
{%- elif 'items' in spec -%}
|
| 570 |
+
{{ render_markdown_detail(indent, "Items", spec['items']) }}
|
| 571 |
+
{%- endif -%}
|
| 572 |
+
{%- if spec.oneOf -%}
|
| 573 |
+
{{- "\n" + indent + " - oneOf:" }}
|
| 574 |
+
{%- for variant in spec.oneOf -%}
|
| 575 |
+
{{- "\n" + indent + " - Variant " }}{{ loop.index }}{{- " *(" + render_markdown_type(variant) + ")*" }}
|
| 576 |
+
{{- render_markdown_schema_details(variant, indent + " ", true) }}
|
| 577 |
+
{%- endfor -%}
|
| 578 |
+
{%- endif -%}
|
| 579 |
+
{%- if spec.anyOf -%}
|
| 580 |
+
{{- "\n" + indent + " - anyOf:" }}
|
| 581 |
+
{%- for variant in spec.anyOf -%}
|
| 582 |
+
{{- "\n" + indent + " - Variant " }}{{ loop.index }}{{- " *(" + render_markdown_type(variant) + ")*" }}
|
| 583 |
+
{{- render_markdown_schema_details(variant, indent + " ", true) }}
|
| 584 |
+
{%- endfor -%}
|
| 585 |
+
{%- endif -%}
|
| 586 |
+
{%- if spec.patternProperties is mapping -%}
|
| 587 |
+
{{- "\n" + indent + " - Pattern properties:" }}
|
| 588 |
+
{%- for pattern, pattern_spec in spec.patternProperties | items -%}
|
| 589 |
+
{%- if pattern_spec is mapping -%}
|
| 590 |
+
{{- "\n" + indent + " - `" + pattern + "` *(" + render_markdown_type(pattern_spec) + ")*" }}
|
| 591 |
+
{{- render_markdown_schema_details(pattern_spec, indent + " ", true) }}
|
| 592 |
+
{%- else -%}
|
| 593 |
+
{{- "\n" + indent + " - `" + pattern + "`: " }}{{ render_markdown_value(pattern_spec) }}
|
| 594 |
+
{%- endif -%}
|
| 595 |
+
{%- endfor -%}
|
| 596 |
+
{%- elif spec.patternProperties is defined -%}
|
| 597 |
+
{{ render_markdown_detail(indent, "Pattern properties", spec.patternProperties) }}
|
| 598 |
+
{%- endif -%}
|
| 599 |
+
{%- if spec.returns is mapping -%}
|
| 600 |
+
{{- "\n" + indent + " - Returns *(" + render_markdown_type(spec.returns) + ")*" }}
|
| 601 |
+
{{- render_markdown_schema_details(spec.returns, indent + " ", true) }}
|
| 602 |
+
{%- elif spec.returns is defined -%}
|
| 603 |
+
{{ render_markdown_detail(indent, "Returns", spec.returns) }}
|
| 604 |
+
{%- endif -%}
|
| 605 |
+
{%- endmacro -%}
|
| 606 |
+
|
| 607 |
+
{%- macro render_markdown_schema_details(spec, indent, include_value_details) -%}
|
| 608 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 609 |
+
{%- set _r = spec['$ref'] -%}
|
| 610 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 611 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 612 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 613 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 614 |
+
{%- if spec['$ref'] is string -%}
|
| 615 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 616 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 617 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 618 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 619 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 620 |
+
{%- endif -%}
|
| 621 |
+
{%- endif -%}
|
| 622 |
+
{%- endif -%}
|
| 623 |
+
{%- endif -%}
|
| 624 |
+
{%- if spec is mapping -%}
|
| 625 |
+
{{- render_markdown_schema_annotations(spec, indent, include_value_details) }}
|
| 626 |
+
{{- render_markdown_schema_structure(spec, indent, true) }}
|
| 627 |
+
{{- render_markdown_schema_extras(spec, indent) }}
|
| 628 |
+
{%- elif spec is not sameas true and spec is not sameas false -%}
|
| 629 |
+
{{- "\n" + indent + " - Value: " }}{{ render_markdown_literal(spec) }}
|
| 630 |
+
{%- endif -%}
|
| 631 |
+
{%- endmacro -%}
|
| 632 |
+
|
| 633 |
+
{%- macro render_markdown_parameter_schema(spec) -%}
|
| 634 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 635 |
+
{%- set _r = spec['$ref'] -%}
|
| 636 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 637 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 638 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 639 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 640 |
+
{%- if spec['$ref'] is string -%}
|
| 641 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 642 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 643 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 644 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 645 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 646 |
+
{%- endif -%}
|
| 647 |
+
{%- endif -%}
|
| 648 |
+
{%- endif -%}
|
| 649 |
+
{%- endif -%}
|
| 650 |
+
{%- if spec is mapping -%}
|
| 651 |
+
{{- render_markdown_metadata_annotations(spec) }}
|
| 652 |
+
{%- if 'items' in spec and spec['items'] is mapping -%}
|
| 653 |
+
{{- "\n- Items *(" + render_markdown_type(spec['items']) + ")*" }}
|
| 654 |
+
{{- render_markdown_schema_details(spec['items'], "", true) }}
|
| 655 |
+
{%- elif 'items' in spec -%}
|
| 656 |
+
{{ render_markdown_metadata_detail("Items", spec['items']) }}
|
| 657 |
+
{%- endif -%}
|
| 658 |
+
{%- if spec.oneOf -%}
|
| 659 |
+
{{- "\n- oneOf:" }}
|
| 660 |
+
{%- for variant in spec.oneOf -%}
|
| 661 |
+
{{- "\n - Variant " }}{{ loop.index }}{{- " *(" + render_markdown_type(variant) + ")*" }}
|
| 662 |
+
{{- render_markdown_schema_details(variant, " ", true) }}
|
| 663 |
+
{%- endfor -%}
|
| 664 |
+
{%- endif -%}
|
| 665 |
+
{%- if spec.anyOf -%}
|
| 666 |
+
{{- "\n- anyOf:" }}
|
| 667 |
+
{%- for variant in spec.anyOf -%}
|
| 668 |
+
{{- "\n - Variant " }}{{ loop.index }}{{- " *(" + render_markdown_type(variant) + ")*" }}
|
| 669 |
+
{{- render_markdown_schema_details(variant, " ", true) }}
|
| 670 |
+
{%- endfor -%}
|
| 671 |
+
{%- endif -%}
|
| 672 |
+
{%- if spec.patternProperties is mapping -%}
|
| 673 |
+
{{- "\n- Pattern properties:" }}
|
| 674 |
+
{%- for pattern, pattern_spec in spec.patternProperties | items -%}
|
| 675 |
+
{%- if pattern_spec is mapping -%}
|
| 676 |
+
{{- "\n - `" + pattern + "` *(" + render_markdown_type(pattern_spec) + ")*" }}
|
| 677 |
+
{{- render_markdown_schema_details(pattern_spec, " ", true) }}
|
| 678 |
+
{%- else -%}
|
| 679 |
+
{{- "\n - `" + pattern + "`: " }}{{ render_markdown_value(pattern_spec) }}
|
| 680 |
+
{%- endif -%}
|
| 681 |
+
{%- endfor -%}
|
| 682 |
+
{%- elif spec.patternProperties is defined -%}
|
| 683 |
+
{{ render_markdown_metadata_detail("Pattern properties", spec.patternProperties) }}
|
| 684 |
+
{%- endif -%}
|
| 685 |
+
{%- if spec.returns is mapping -%}
|
| 686 |
+
{{- "\n- Returns *(" + render_markdown_type(spec.returns) + ")*" }}
|
| 687 |
+
{{- render_markdown_schema_details(spec.returns, "", true) }}
|
| 688 |
+
{%- elif spec.returns is defined -%}
|
| 689 |
+
{{ render_markdown_metadata_detail("Returns", spec.returns) }}
|
| 690 |
+
{%- endif -%}
|
| 691 |
+
{{- render_markdown_metadata_extras(spec) }}
|
| 692 |
+
{%- endif -%}
|
| 693 |
+
{%- endmacro -%}
|
| 694 |
+
|
| 695 |
+
{%- macro render_markdown_param(name, spec, required_list, indent) -%}
|
| 696 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 697 |
+
{%- set _r = spec['$ref'] -%}
|
| 698 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 699 |
+
{%- if _k is not none and ('|' + _k + '|') not in REFS.seen and REFS.defs[_k] is mapping -%}
|
| 700 |
+
{%- set spec = dict((REFS.defs[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 701 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 702 |
+
{%- if spec['$ref'] is string -%}
|
| 703 |
+
{%- set _r2 = spec['$ref'] -%}
|
| 704 |
+
{%- set _k2 = _r2[8:] if _r2.startswith('#/$defs/') else (_r2[14:] if _r2.startswith('#/definitions/') else none) -%}
|
| 705 |
+
{%- if _k2 is not none and REFS.defs[_k2] is mapping -%}
|
| 706 |
+
{%- set spec = dict((REFS.defs[_k2] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 707 |
+
{%- set REFS.seen = REFS.seen + _k2 + '|' -%}
|
| 708 |
+
{%- endif -%}
|
| 709 |
+
{%- endif -%}
|
| 710 |
+
{%- endif -%}
|
| 711 |
+
{%- endif -%}
|
| 712 |
+
{{- "\n" + indent + "- `" + name + "` *(" + render_markdown_type(spec) }}
|
| 713 |
+
{%- if name in (required_list or []) -%}{{- ", required" }}{%- endif -%}
|
| 714 |
+
{{- ")*" }}
|
| 715 |
+
{%- if spec.description -%}{{- " - " + spec.description | replace("\n", "\n" + indent + " ") }}{%- endif -%}
|
| 716 |
+
{%- if spec.enum -%}
|
| 717 |
+
{{- "\n" + indent + " - Allowed values: " }}{{ render_allowed_values(spec.enum) }}
|
| 718 |
+
{%- endif -%}
|
| 719 |
+
{%- if spec.default is defined -%}
|
| 720 |
+
{{- "\n" + indent + " - Default: " }}{{ render_markdown_literal(spec.default) }}
|
| 721 |
+
{%- endif -%}
|
| 722 |
+
{{- render_markdown_schema_details(spec, indent, false) }}
|
| 723 |
+
{%- endmacro -%}
|
| 724 |
+
|
| 725 |
+
{%- macro render_tools_markdown(tools_list) -%}
|
| 726 |
+
{{- "<ifm|tools>" }}
|
| 727 |
+
{%- for tool in tools_list -%}
|
| 728 |
+
{%- set fn = tool.function if tool.function is defined else tool -%}
|
| 729 |
+
{%- set REFS.defs = fn.parameters['$defs'] if (fn.parameters is mapping and fn.parameters['$defs'] is mapping) else (fn.parameters['definitions'] if (fn.parameters is mapping and fn.parameters['definitions'] is mapping) else {}) -%}
|
| 730 |
+
{%- set REFS.seen = '|' -%}
|
| 731 |
+
{%- set fnp = namespace(p=fn.parameters) -%}
|
| 732 |
+
{%- if fnp.p is mapping and fnp.p['$ref'] is string -%}
|
| 733 |
+
{%- set _r = fnp.p['$ref'] -%}
|
| 734 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 735 |
+
{%- if _k is not none and REFS.defs[_k] is mapping -%}
|
| 736 |
+
{%- set fnp.p = dict((REFS.defs[_k] | items | list) + (fnp.p | items | rejectattr('0', 'equalto', '$ref') | list)) -%}
|
| 737 |
+
{%- set REFS.seen = REFS.seen + _k + '|' -%}
|
| 738 |
+
{%- endif -%}
|
| 739 |
+
{%- endif -%}
|
| 740 |
+
{{- "\n## " + fn.name }}
|
| 741 |
+
{%- if fn.description -%}
|
| 742 |
+
{{- "\n" + fn.description }}
|
| 743 |
+
{%- endif -%}
|
| 744 |
+
{{- "\n\n**Parameters**" }}
|
| 745 |
+
{%- if fnp.p and fnp.p.properties -%}
|
| 746 |
+
{%- for pname, pspec in fnp.p.properties | items -%}
|
| 747 |
+
{{- render_markdown_param(pname, pspec, fnp.p.required or [], "") }}
|
| 748 |
+
{%- endfor -%}
|
| 749 |
+
{%- elif fnp.p is mapping and (fnp.p.oneOf or fnp.p.anyOf or 'items' in fnp.p) -%}
|
| 750 |
+
{{- render_markdown_parameter_schema(fnp.p) }}
|
| 751 |
+
{%- else -%}
|
| 752 |
+
{{- "\n- None" }}
|
| 753 |
+
{%- endif -%}
|
| 754 |
+
{%- set fn_ret = fn.returns if fn.returns is defined else fn.response -%}
|
| 755 |
+
{%- if fn_ret is mapping -%}
|
| 756 |
+
{{- "\n\n**Returns**" }}
|
| 757 |
+
{{- "\n- Return *(" + render_markdown_type(fn_ret) + ")*" }}
|
| 758 |
+
{{- render_markdown_schema_details(fn_ret, "", true) }}
|
| 759 |
+
{%- elif fn_ret is defined -%}
|
| 760 |
+
{{- "\n\n**Returns**\n- " }}{{ render_markdown_value(fn_ret) }}
|
| 761 |
+
{%- endif -%}
|
| 762 |
+
{%- if not loop.last -%}{{- "\n" }}{%- endif -%}
|
| 763 |
+
{%- endfor -%}
|
| 764 |
+
{{- "\n</ifm|tools>" }}
|
| 765 |
+
{%- endmacro -%}
|
| 766 |
+
|
| 767 |
+
{%- macro render_tool_presentation(tools_list, fmt) -%}
|
| 768 |
+
{%- if fmt == 'json' -%}
|
| 769 |
+
{{- render_tools_json(tools_list) }}
|
| 770 |
+
{%- elif RB.bad != '|' -%}
|
| 771 |
+
{#- some tool uses constructs the pretty renderers cannot represent (verdicts -#}
|
| 772 |
+
{#- computed during validate_tools): render the WHOLE toolset exactly as the -#}
|
| 773 |
+
{#- json presentation would, so the block stays uniform and model-familiar. -#}
|
| 774 |
+
{{- render_tools_json(tools_list) }}
|
| 775 |
+
{%- elif fmt == 'xml' -%}
|
| 776 |
+
{{- render_tools_xml(tools_list) }}
|
| 777 |
+
{%- elif fmt == 'markdown' -%}
|
| 778 |
+
{{- render_tools_markdown(tools_list) }}
|
| 779 |
+
{%- else -%}
|
| 780 |
+
{{- raise_exception("Unsupported tool_presentation_format: '" + fmt + "'. Supported formats: json, xml, markdown.") }}
|
| 781 |
+
{%- endif -%}
|
| 782 |
+
{%- endmacro -%}
|
| 783 |
+
|
| 784 |
+
{%- macro render_call_instructions(fmt) -%}
|
| 785 |
+
{%- if fmt == 'json' -%}
|
| 786 |
+
{{- "Wrap all tool calls in a single <ifm|tool_calls></ifm|tool_calls> block. For each call, emit one JSON object with the function name and arguments on the same line inside <ifm|tool_call></ifm|tool_call> tags:\n\n<ifm|tool_calls>\n<ifm|tool_call>{\"name\": <function-name>, \"arguments\": <args-json-object>}</ifm|tool_call>\n</ifm|tool_calls>" }}
|
| 787 |
+
{%- elif fmt == 'xml' -%}
|
| 788 |
+
{{- "Wrap all tool calls in a single <ifm|tool_calls></ifm|tool_calls> block. For each call, write the function name at the start of <ifm|tool_call>, followed by paired <ifm|arg_key> and <ifm|arg_value> tags for each argument:\n\n<ifm|tool_calls>\n<ifm|tool_call>$FUNCTION_NAME\n<ifm|arg_key>$PARAMETER_NAME</ifm|arg_key>\n<ifm|arg_value>$PARAMETER_VALUE</ifm|arg_value>\n...\n</ifm|tool_call>\n</ifm|tool_calls>\n\nString and scalar parameters should be written as plain text. Array and object parameters should be written as JSON literals." }}
|
| 789 |
+
{%- elif fmt == 'xml_typed' -%}
|
| 790 |
+
{{- "Wrap all tool calls in a single <ifm|tool_calls></ifm|tool_calls> block. For each call, write the function name at the start of <ifm|tool_call>, followed by <ifm|arg_key>, <ifm|arg_type>, and <ifm|arg_value> tags for each argument:\n\n<ifm|tool_calls>\n<ifm|tool_call>$FUNCTION_NAME\n<ifm|arg_key>$PARAMETER_NAME</ifm|arg_key>\n<ifm|arg_type>$ARGUMENT_TYPE</ifm|arg_type>\n<ifm|arg_value>$PARAMETER_VALUE</ifm|arg_value>\n...\n</ifm|tool_call>\n</ifm|tool_calls>\n\nUse the parameter type shown in the tool definition. If that type contains anyOf or oneOf, use the actual argument value type instead. String and scalar parameters should be written as plain text. Array and object parameters should be written as JSON literals." }}
|
| 791 |
+
{%- else -%}
|
| 792 |
+
{{- raise_exception("Unsupported tool_call_format: '" + fmt + "'. Supported formats: json, xml, xml_typed.") }}
|
| 793 |
+
{%- endif -%}
|
| 794 |
+
{%- endmacro -%}
|
| 795 |
+
|
| 796 |
+
{%- macro render_system_with_tools(tools_list, system_content, presentation_fmt, call_fmt) -%}
|
| 797 |
+
{{- "<|ifm|im_start|>system\n# Tools\nYou may call one or more tools to assist with the user query.\n\nAvailable tools are:\n\n" }}
|
| 798 |
+
{{- render_tool_presentation(tools_list, presentation_fmt) }}
|
| 799 |
+
{{- "\n\nWhen calling tools, you MUST follow the tool-call format below:\n\n" }}
|
| 800 |
+
{{- render_call_instructions(call_fmt) }}
|
| 801 |
+
{%- if system_content -%}
|
| 802 |
+
{{- "\n\n" + system_content }}
|
| 803 |
+
{%- endif -%}
|
| 804 |
+
{{- "<|ifm|im_end|>" }}
|
| 805 |
+
{%- endmacro -%}
|
| 806 |
+
|
| 807 |
+
{%- macro render_argument_value(value) -%}
|
| 808 |
+
{%- if value is string -%}{{- value -}}{%- else -%}{{- value | tojson -}}{%- endif -%}
|
| 809 |
+
{%- endmacro -%}
|
| 810 |
+
|
| 811 |
+
{%- macro render_value_type(value) -%}
|
| 812 |
+
{%- if value is none -%}null
|
| 813 |
+
{%- elif value is boolean -%}boolean
|
| 814 |
+
{%- elif value is integer -%}integer
|
| 815 |
+
{%- elif value is number -%}number
|
| 816 |
+
{%- elif value is string -%}string
|
| 817 |
+
{%- elif value is mapping -%}object
|
| 818 |
+
{%- elif value is sequence -%}array
|
| 819 |
+
{%- else -%}any
|
| 820 |
+
{%- endif -%}
|
| 821 |
+
{%- endmacro -%}
|
| 822 |
+
|
| 823 |
+
{%- macro schema_has_combinator(spec) -%}
|
| 824 |
+
{%- if spec.oneOf or spec.anyOf -%}
|
| 825 |
+
true
|
| 826 |
+
{%- elif spec.type is defined and spec.type is sequence and spec.type is not string and spec.type | length > 1 -%}
|
| 827 |
+
true
|
| 828 |
+
{%- elif spec.type == "array" and 'items' in spec -%}
|
| 829 |
+
{{- schema_has_combinator(spec['items']) -}}
|
| 830 |
+
{%- elif spec.properties -%}
|
| 831 |
+
{%- set found = namespace(value='false') -%}
|
| 832 |
+
{%- for child_name, child_spec in spec.properties | items -%}
|
| 833 |
+
{%- if schema_has_combinator(child_spec) == 'true' -%}
|
| 834 |
+
{%- set found.value = 'true' -%}
|
| 835 |
+
{%- endif -%}
|
| 836 |
+
{%- endfor -%}
|
| 837 |
+
{{- found.value -}}
|
| 838 |
+
{%- else -%}
|
| 839 |
+
false
|
| 840 |
+
{%- endif -%}
|
| 841 |
+
{%- endmacro -%}
|
| 842 |
+
|
| 843 |
+
{%- macro render_arg_type(tools_list, tool_name, arg_name, value) -%}
|
| 844 |
+
{%- set found = namespace(type='any') -%}
|
| 845 |
+
{%- for tool in tools_list -%}
|
| 846 |
+
{%- set fn = tool.function if tool.function is defined else tool -%}
|
| 847 |
+
{%- if fn.name == tool_name and fn.parameters and fn.parameters.properties and arg_name in fn.parameters.properties -%}
|
| 848 |
+
{%- set spec = fn.parameters.properties[arg_name] -%}
|
| 849 |
+
{%- if spec is mapping and spec['$ref'] is string -%}
|
| 850 |
+
{%- set _r = spec['$ref'] -%}
|
| 851 |
+
{%- set _k = _r[8:] if _r.startswith('#/$defs/') else (_r[14:] if _r.startswith('#/definitions/') else none) -%}
|
| 852 |
+
{%- set _d = fn.parameters['$defs'] if fn.parameters['$defs'] is mapping else fn.parameters['definitions'] -%}
|
| 853 |
+
{%- set spec = dict((_d[_k] | items | list) + (spec | items | rejectattr('0', 'equalto', '$ref') | list)) if (_k is not none and _d is mapping and _d[_k] is mapping) else spec -%}
|
| 854 |
+
{%- endif -%}
|
| 855 |
+
{%- if schema_has_combinator(spec) == 'true' -%}
|
| 856 |
+
{%- set found.type = render_value_type(value) -%}
|
| 857 |
+
{%- else -%}
|
| 858 |
+
{%- set found.type = render_compact_type(spec) -%}
|
| 859 |
+
{%- endif -%}
|
| 860 |
+
{%- endif -%}
|
| 861 |
+
{%- endfor -%}
|
| 862 |
+
{{- found.type -}}
|
| 863 |
+
{%- endmacro -%}
|
| 864 |
+
|
| 865 |
+
{%- macro render_tool_calls_block(tool_calls, fmt, tools_list) -%}
|
| 866 |
+
{{- "<ifm|tool_calls>" }}
|
| 867 |
+
{%- for raw_tool_call in tool_calls -%}
|
| 868 |
+
{%- set tool_call = raw_tool_call.function if raw_tool_call.function else raw_tool_call -%}
|
| 869 |
+
{%- if tool_call.arguments is string -%}
|
| 870 |
+
{{- raise_exception("tool_call.arguments must be a dict, not a JSON string. Parse it before passing to the template.") -}}
|
| 871 |
+
{%- endif -%}
|
| 872 |
+
{%- if fmt == 'json' -%}
|
| 873 |
+
{{- "\n<ifm|tool_call>{\"name\": \"" + tool_call.name + "\", \"arguments\": " }}{{ tool_call.arguments | tojson }}{{- "}</ifm|tool_call>" }}
|
| 874 |
+
{%- elif fmt == 'xml' or fmt == 'xml_typed' -%}
|
| 875 |
+
{{- "\n<ifm|tool_call>" + tool_call.name + "\n" }}
|
| 876 |
+
{%- for key, value in tool_call.arguments | items -%}
|
| 877 |
+
{{- "<ifm|arg_key>" + key + "</ifm|arg_key>\n" }}
|
| 878 |
+
{%- if fmt == 'xml_typed' -%}
|
| 879 |
+
{{- "<ifm|arg_type>" + render_arg_type(tools_list, tool_call.name, key, value) + "</ifm|arg_type>\n" }}
|
| 880 |
+
{%- endif -%}
|
| 881 |
+
{{- "<ifm|arg_value>" }}{{ render_argument_value(value) }}{{- "</ifm|arg_value>\n" }}
|
| 882 |
+
{%- endfor -%}
|
| 883 |
+
{{- "</ifm|tool_call>" }}
|
| 884 |
+
{%- else -%}
|
| 885 |
+
{{- raise_exception("Unsupported tool_call_format: '" + fmt + "'. Supported formats: json, xml, xml_typed.") -}}
|
| 886 |
+
{%- endif -%}
|
| 887 |
+
{%- endfor -%}
|
| 888 |
+
{{- "\n</ifm|tool_calls>" }}
|
| 889 |
+
{%- endmacro -%}
|
| 890 |
+
|
| 891 |
+
{%- macro render_tool_response_messages(raw_content) -%}
|
| 892 |
+
{%- if raw_content is string -%}
|
| 893 |
+
{{- '<|ifm|im_start|>tool\n' + raw_content + '<|ifm|im_end|>' }}
|
| 894 |
+
{%- elif raw_content is sequence and raw_content is not string and raw_content is not mapping -%}
|
| 895 |
+
{%- if raw_content | length == 0 -%}
|
| 896 |
+
{{- raise_exception("tool message content list must not be empty.") -}}
|
| 897 |
+
{%- endif -%}
|
| 898 |
+
{{- '<|ifm|im_start|>tool\n' -}}
|
| 899 |
+
{%- for item in raw_content -%}
|
| 900 |
+
{%- if not loop.first -%}{{- '\n' -}}{%- endif -%}
|
| 901 |
+
{%- if item is string -%}
|
| 902 |
+
{{- item -}}
|
| 903 |
+
{%- elif item is mapping and item.text is string -%}
|
| 904 |
+
{{- item.text -}}
|
| 905 |
+
{%- else -%}
|
| 906 |
+
{{- (item | tojson) -}}
|
| 907 |
+
{%- endif -%}
|
| 908 |
+
{%- endfor -%}
|
| 909 |
+
{{- '<|ifm|im_end|>' -}}
|
| 910 |
+
{%- else -%}
|
| 911 |
+
{{- '<|ifm|im_start|>tool\n' }}{{ raw_content | tojson }}{{- '<|ifm|im_end|>' }}
|
| 912 |
+
{%- endif -%}
|
| 913 |
+
{%- endmacro -%}
|
| 914 |
+
|
| 915 |
+
{%- set available_tools = tools if tools else [] -%}
|
| 916 |
+
{%- if (not available_tools) and messages[0].role == 'system' and messages[0].get('tools') -%}
|
| 917 |
+
{%- set available_tools = messages[0]['tools'] -%}
|
| 918 |
+
{%- endif -%}
|
| 919 |
+
{%- if available_tools -%}
|
| 920 |
+
{{- validate_tools(available_tools, tool_presentation_fmt != 'json') }}
|
| 921 |
+
{%- set system_content = '' -%}
|
| 922 |
+
{%- if messages[0].role == 'system' and messages[0].content -%}
|
| 923 |
+
{%- set system_content = messages[0].content -%}
|
| 924 |
+
{%- endif -%}
|
| 925 |
+
{{- render_system_with_tools(available_tools, system_content, tool_presentation_fmt, tool_call_fmt) }}
|
| 926 |
+
{%- else -%}
|
| 927 |
+
{%- if messages[0].role == 'system' -%}
|
| 928 |
+
{{- '<|ifm|im_start|>system\n' + messages[0].content + '<|ifm|im_end|>' }}
|
| 929 |
+
{%- endif -%}
|
| 930 |
+
{%- endif -%}
|
| 931 |
+
|
| 932 |
+
{%- for message in messages -%}
|
| 933 |
+
{%- if message.content is string -%}
|
| 934 |
+
{%- set content = message.content -%}
|
| 935 |
+
{%- else -%}
|
| 936 |
+
{%- set content = '' -%}
|
| 937 |
+
{%- endif -%}
|
| 938 |
+
{%- if (message.role == "user") or (message.role == "system" and not loop.first) -%}
|
| 939 |
+
{{- '<|ifm|im_start|>' + message.role + '\n' + content + '<|ifm|im_end|>' }}
|
| 940 |
+
{%- elif message.role == "assistant" -%}
|
| 941 |
+
{%- set thinking_content = '' -%}
|
| 942 |
+
{%- set think_tag = 'ifm|think' -%}
|
| 943 |
+
{%- if message.think is defined and message.think is string -%}
|
| 944 |
+
{%- set thinking_content = message.think -%}
|
| 945 |
+
{%- set think_tag = 'ifm|think' -%}
|
| 946 |
+
{%- elif message.think_fast is defined and message.think_fast is string -%}
|
| 947 |
+
{%- set thinking_content = message.think_fast -%}
|
| 948 |
+
{%- set think_tag = 'ifm|think_fast' -%}
|
| 949 |
+
{%- elif message.think_faster is defined and message.think_faster is string -%}
|
| 950 |
+
{%- set thinking_content = message.think_faster -%}
|
| 951 |
+
{%- set think_tag = 'ifm|think_faster' -%}
|
| 952 |
+
{%- elif message.reasoning_content is defined and message.reasoning_content is string -%}
|
| 953 |
+
{%- set thinking_content = message.reasoning_content -%}
|
| 954 |
+
{%- set think_tag = 'ifm|think' -%}
|
| 955 |
+
{%- elif message.reasoning is defined and message.reasoning is string -%}
|
| 956 |
+
{%- set thinking_content = message.reasoning -%}
|
| 957 |
+
{%- set think_tag = 'ifm|think' -%}
|
| 958 |
+
{%- elif message.think is not defined and message.reasoning is not defined and message.reasoning_content is not defined and message.think_fast is not defined and message.think_faster is not defined -%}
|
| 959 |
+
{#- MLX: tolerate assistant history without a thinking field (render empty thinking) instead of raising. -#}
|
| 960 |
+
{%- else -%}
|
| 961 |
+
{{- raise_exception("Assistant thinking fields must be strings. Provide one of: think, reasoning, reasoning_content, think_fast, think_faster as a string.") -}}
|
| 962 |
+
{%- endif -%}
|
| 963 |
+
{{- '<|ifm|im_start|>' + message.role }}
|
| 964 |
+
{% generation %}
|
| 965 |
+
{%- if think_tag -%}
|
| 966 |
+
{%- if thinking_content -%}
|
| 967 |
+
{{- '<' + think_tag + '>\n' + thinking_content + '</' + think_tag + '>' + content }}
|
| 968 |
+
{%- else -%}
|
| 969 |
+
{{- '<' + think_tag + '>\n</' + think_tag + '>' + content }}
|
| 970 |
+
{%- endif -%}
|
| 971 |
+
{%- else -%}
|
| 972 |
+
{{- content }}
|
| 973 |
+
{%- endif -%}
|
| 974 |
+
{%- if message.tool_calls -%}
|
| 975 |
+
{{- render_tool_calls_block(message.tool_calls, tool_call_fmt, available_tools) }}
|
| 976 |
+
{%- endif -%}
|
| 977 |
+
{{- '<|ifm|im_end|>' -}}
|
| 978 |
+
{%- endgeneration -%}
|
| 979 |
+
{%- elif message.role == "tool" -%}
|
| 980 |
+
{{- render_tool_response_messages(message.content) }}
|
| 981 |
+
{%- endif -%}
|
| 982 |
+
{%- endfor -%}
|
| 983 |
+
{%- if add_generation_prompt -%}
|
| 984 |
+
{%- set effort = reasoning_effort | default('high') -%}
|
| 985 |
+
{%- if effort == 'high' -%}
|
| 986 |
+
{{- '<|ifm|im_start|>assistant\n<ifm|think>\n' }}
|
| 987 |
+
{%- elif effort == 'medium' -%}
|
| 988 |
+
{{- '<|ifm|im_start|>assistant\n<ifm|think_fast>\n' }}
|
| 989 |
+
{%- elif effort == 'low' -%}
|
| 990 |
+
{{- '<|ifm|im_start|>assistant\n<ifm|think_faster>\n' }}
|
| 991 |
+
{%- else -%}
|
| 992 |
+
{{- raise_exception("Unsupported reasoning_effort: '" + effort + "'. Supported values: high, medium, low.") -}}
|
| 993 |
+
{%- endif -%}
|
| 994 |
+
{%- endif -%}
|
config.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
generation_config.json
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 0,
|
| 3 |
+
"do_sample": false,
|
| 4 |
+
"eos_token_id": [
|
| 5 |
+
1,
|
| 6 |
+
250019
|
| 7 |
+
]
|
| 8 |
+
}
|
k2_horizon.py
ADDED
|
@@ -0,0 +1,628 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright © 2026 Apple Inc.
|
| 2 |
+
|
| 3 |
+
"""K2-Horizon for MLX: MoE with Mixture-of-Values (MoVA) attention.
|
| 4 |
+
|
| 5 |
+
This file is an MLX port of the official PyTorch implementation,
|
| 6 |
+
``modeling_k2_horizon.py`` in IFM/K2-Horizon-MoVA-36B-A4B (revision e5c131d).
|
| 7 |
+
The math is the same. The implementation is different in these ways:
|
| 8 |
+
|
| 9 |
+
1. Experts are computed with ``gather_mm`` (``SwitchGLU`` / ``SwitchLinear``).
|
| 10 |
+
The reference loops over experts in Python, finds each expert's tokens with
|
| 11 |
+
``one_hot`` / ``torch.where`` and accumulates with ``index_add_``. That style
|
| 12 |
+
needs in-place updates and data-dependent control flow, which are slow or not
|
| 13 |
+
available in MLX's lazy graph, so all selected experts run in one kernel call.
|
| 14 |
+
2. Per-expert weights are stacked into one tensor per projection when the
|
| 15 |
+
checkpoint loads (see ``Model.sanitize``).
|
| 16 |
+
3. ``K2HorizonAttention`` and ``K2HorizonMoVAAttention`` are merged into one
|
| 17 |
+
``Attention`` class. Only the source of the values differs.
|
| 18 |
+
4. Attention, RoPE and RMSNorm use MLX fused kernels (``mx.fast.*``) instead of
|
| 19 |
+
the eager PyTorch ops.
|
| 20 |
+
5. In bf16, expert outputs are weighted and summed in fp32. The reference casts
|
| 21 |
+
the routing weights to bf16 and accumulates in bf16. The result is slightly
|
| 22 |
+
more precise, so bf16 outputs differ from bf16 PyTorch by rounding only.
|
| 23 |
+
6. KV cache and masks use mlx-lm's cache objects, not the HF ``Cache`` classes.
|
| 24 |
+
|
| 25 |
+
Checked against the reference in fp32 on the real weights: every layer matches
|
| 26 |
+
to a relative error of 1e-6 or better, and greedy decoding with the KV cache
|
| 27 |
+
gives the same tokens.
|
| 28 |
+
|
| 29 |
+
Not supported (the reference supports them, this checkpoint does not use them):
|
| 30 |
+
partial RoPE (``rope_head_dim != head_dim``), sliding-window attention, and the
|
| 31 |
+
training-only parts (dropout, router logits output, load-balancing loss).
|
| 32 |
+
"""
|
| 33 |
+
|
| 34 |
+
import math
|
| 35 |
+
from dataclasses import dataclass
|
| 36 |
+
from typing import Any, Dict, List, Optional
|
| 37 |
+
|
| 38 |
+
import mlx.core as mx
|
| 39 |
+
import mlx.nn as nn
|
| 40 |
+
|
| 41 |
+
from mlx_lm.models.activations import swiglu
|
| 42 |
+
from mlx_lm.models.base import BaseModelArgs, create_attention_mask, scaled_dot_product_attention
|
| 43 |
+
from mlx_lm.models.rope_utils import initialize_rope
|
| 44 |
+
from mlx_lm.models.switch_layers import SwitchGLU, SwitchLinear, _gather_sort, _scatter_unsort
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@dataclass
|
| 48 |
+
class ModelArgs(BaseModelArgs):
|
| 49 |
+
"""Model config. Same fields and defaults as ``K2HorizonConfig``.
|
| 50 |
+
|
| 51 |
+
Fields that only matter for training (``router_aux_loss_coef``,
|
| 52 |
+
``attention_dropout``, ``output_router_logits``) are ignored.
|
| 53 |
+
"""
|
| 54 |
+
|
| 55 |
+
model_type: str
|
| 56 |
+
hidden_size: int
|
| 57 |
+
num_hidden_layers: int
|
| 58 |
+
intermediate_size: int
|
| 59 |
+
num_attention_heads: int
|
| 60 |
+
num_key_value_heads: int
|
| 61 |
+
head_dim: int
|
| 62 |
+
vocab_size: int
|
| 63 |
+
rms_norm_eps: float
|
| 64 |
+
max_position_embeddings: int
|
| 65 |
+
num_experts: int = 0
|
| 66 |
+
num_experts_per_tok: int = 0
|
| 67 |
+
moe_intermediate_size: int = 0
|
| 68 |
+
decoder_sparse_step: int = 1
|
| 69 |
+
mlp_only_layers: Optional[List[int]] = None
|
| 70 |
+
norm_topk_prob: bool = False
|
| 71 |
+
num_shared_experts: int = 0
|
| 72 |
+
moe_gate_bias: bool = False
|
| 73 |
+
router_score_func: str = "softmax"
|
| 74 |
+
router_scaling_factor: Optional[float] = 1.0
|
| 75 |
+
mova_num_experts: int = 0
|
| 76 |
+
mova_num_experts_per_tok: int = 0
|
| 77 |
+
attention_bias: bool = False
|
| 78 |
+
attention_gate_func: Optional[str] = None
|
| 79 |
+
query_key_norm: bool = True
|
| 80 |
+
layernorm_num_groups: int = 1
|
| 81 |
+
rope_head_dim: Optional[int] = None
|
| 82 |
+
rope_parameters: Optional[Dict[str, Any]] = None
|
| 83 |
+
rope_theta: float = 10000.0
|
| 84 |
+
use_sliding_window: bool = False
|
| 85 |
+
tie_word_embeddings: bool = False
|
| 86 |
+
|
| 87 |
+
def __post_init__(self):
|
| 88 |
+
# Same defaults as K2HorizonConfig.__post_init__.
|
| 89 |
+
if self.mlp_only_layers is None:
|
| 90 |
+
self.mlp_only_layers = []
|
| 91 |
+
if self.router_scaling_factor is None:
|
| 92 |
+
self.router_scaling_factor = 1.0
|
| 93 |
+
# Newer configs keep rope_theta inside rope_parameters.
|
| 94 |
+
if self.rope_parameters is not None:
|
| 95 |
+
self.rope_theta = self.rope_parameters.get("rope_theta", self.rope_theta)
|
| 96 |
+
# The reference supports these two features. They are not ported, so fail
|
| 97 |
+
# loudly instead of computing wrong outputs.
|
| 98 |
+
if self.rope_head_dim is not None and self.rope_head_dim != self.head_dim:
|
| 99 |
+
raise NotImplementedError("Partial RoPE is not supported for k2_horizon.")
|
| 100 |
+
if self.use_sliding_window:
|
| 101 |
+
raise NotImplementedError("Sliding window is not supported for k2_horizon.")
|
| 102 |
+
# YaRN RoPE (used by K2-Horizon-0.9B) goes through mlx-lm's YarnRoPE.
|
| 103 |
+
# Check that the config does not use options YarnRoPE handles differently.
|
| 104 |
+
rope = self.rope_parameters or {}
|
| 105 |
+
if rope.get("rope_type", rope.get("type")) == "yarn":
|
| 106 |
+
_check_yarn(rope)
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def _check_yarn(rope):
|
| 110 |
+
"""Reject YaRN configs where mlx-lm's YarnRoPE differs from the reference.
|
| 111 |
+
|
| 112 |
+
The reference uses transformers' ``_compute_yarn_parameters``. mlx-lm's
|
| 113 |
+
``YarnRoPE`` computes the same frequencies, with two differences:
|
| 114 |
+
|
| 115 |
+
- It always truncates the correction range (floor/ceil). The reference does
|
| 116 |
+
that only when ``truncate`` is True, which is the default.
|
| 117 |
+
- It always derives the attention scale from ``factor``, ``mscale`` and
|
| 118 |
+
``mscale_all_dim``. The reference uses ``attention_factor`` directly when
|
| 119 |
+
the config gives one, and treats ``mscale`` differently when
|
| 120 |
+
``mscale_all_dim`` is missing.
|
| 121 |
+
|
| 122 |
+
K2-Horizon-0.9B sets truncate=True and attention_factor=1.27726, which is
|
| 123 |
+
exactly 0.1 * ln(16) + 1, so both agree. Other values raise here instead of
|
| 124 |
+
giving wrong outputs.
|
| 125 |
+
"""
|
| 126 |
+
if not rope.get("truncate", True):
|
| 127 |
+
raise NotImplementedError("YaRN with truncate=False is not supported.")
|
| 128 |
+
if rope.get("factor") is None:
|
| 129 |
+
# The reference would derive factor from max_position_embeddings.
|
| 130 |
+
raise NotImplementedError("YaRN without a factor is not supported.")
|
| 131 |
+
|
| 132 |
+
def get_mscale(scale, m=1):
|
| 133 |
+
return 1.0 if scale <= 1 else 0.1 * m * math.log(scale) + 1.0
|
| 134 |
+
|
| 135 |
+
factor = rope["factor"]
|
| 136 |
+
mscale, mscale_all_dim = rope.get("mscale"), rope.get("mscale_all_dim")
|
| 137 |
+
# The scale YarnRoPE applies (its defaults are mscale=1, mscale_all_dim=0).
|
| 138 |
+
ours = get_mscale(factor, mscale or 1) / get_mscale(factor, mscale_all_dim or 0)
|
| 139 |
+
# The scale the reference applies.
|
| 140 |
+
if rope.get("attention_factor") is not None:
|
| 141 |
+
ref = rope["attention_factor"]
|
| 142 |
+
elif mscale and mscale_all_dim:
|
| 143 |
+
ref = get_mscale(factor, mscale) / get_mscale(factor, mscale_all_dim)
|
| 144 |
+
else:
|
| 145 |
+
ref = get_mscale(factor)
|
| 146 |
+
if abs(ours - ref) > 1e-6:
|
| 147 |
+
raise NotImplementedError(
|
| 148 |
+
f"YaRN attention factor {ref} is not supported (MLX would use {ours})."
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
class GroupedRMSNorm(nn.Module):
|
| 153 |
+
"""RMSNorm applied independently to ``groups`` equal slices of the input.
|
| 154 |
+
|
| 155 |
+
Reference: ``K2HorizonRMSNorm``. Same math: reshape to (..., groups, dims //
|
| 156 |
+
groups), normalize each group in fp32, reshape back, multiply by the
|
| 157 |
+
full-width weight, cast back to the input dtype. The per-group
|
| 158 |
+
normalization uses the fused ``mx.fast.rms_norm`` kernel.
|
| 159 |
+
"""
|
| 160 |
+
|
| 161 |
+
def __init__(self, dims: int, groups: int, eps: float):
|
| 162 |
+
super().__init__()
|
| 163 |
+
self.weight = mx.ones((dims,))
|
| 164 |
+
self.groups = groups
|
| 165 |
+
self.eps = eps
|
| 166 |
+
|
| 167 |
+
def __call__(self, x):
|
| 168 |
+
if self.groups == 1:
|
| 169 |
+
# Fast path: the fused kernel also applies the weight. In bf16 this
|
| 170 |
+
# can round slightly differently from the reference. This checkpoint
|
| 171 |
+
# uses groups > 1 for all its norms, so it does not take this path.
|
| 172 |
+
return mx.fast.rms_norm(x, self.weight, self.eps)
|
| 173 |
+
shape = x.shape
|
| 174 |
+
y = x.astype(mx.float32).reshape(*shape[:-1], self.groups, -1)
|
| 175 |
+
y = mx.fast.rms_norm(y, None, self.eps).reshape(shape)
|
| 176 |
+
return (self.weight * y).astype(x.dtype)
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
class Router(nn.Module):
|
| 180 |
+
"""Top-k router shared by the MoE block and MoVA attention.
|
| 181 |
+
|
| 182 |
+
Reference: ``calc_router_weights`` (used by MoVA) and the routing code at the
|
| 183 |
+
start of ``K2HorizonSparseMoeBlock.forward``. Both do:
|
| 184 |
+
|
| 185 |
+
1. logits = x @ W.T, without the bias (the reference calls ``F.linear`` with
|
| 186 |
+
the weight only)
|
| 187 |
+
2. scores = sigmoid(logits) or softmax(logits), in fp32
|
| 188 |
+
3. select the top-k experts using scores + bias
|
| 189 |
+
4. weights = the *unbiased* scores of the selected experts
|
| 190 |
+
5. optionally renormalize the weights to sum to 1, then multiply by
|
| 191 |
+
``router_scaling_factor``
|
| 192 |
+
|
| 193 |
+
So the bias only changes *which* experts are selected, never the weights.
|
| 194 |
+
Getting this wrong changes the output a lot, and the parity tests check it.
|
| 195 |
+
|
| 196 |
+
Differences from the reference:
|
| 197 |
+
- ``torch.topk`` is replaced by ``mx.argpartition``, which does not sort the
|
| 198 |
+
selected experts. The order does not matter because the outputs are
|
| 199 |
+
summed. Results differ only when two scores are exactly equal.
|
| 200 |
+
- The weights stay in fp32. The MoE reference casts them to the hidden dtype
|
| 201 |
+
(bf16) before use.
|
| 202 |
+
"""
|
| 203 |
+
|
| 204 |
+
def __init__(
|
| 205 |
+
self,
|
| 206 |
+
dims: int,
|
| 207 |
+
num_experts: int,
|
| 208 |
+
top_k: int,
|
| 209 |
+
score_func: str,
|
| 210 |
+
scaling_factor: float,
|
| 211 |
+
renormalize: bool,
|
| 212 |
+
bias: bool,
|
| 213 |
+
):
|
| 214 |
+
super().__init__()
|
| 215 |
+
# Parameter names match the checkpoint: ``mlp.gate.{weight,bias}`` and
|
| 216 |
+
# ``self_attn.v_router.{weight,bias}``. This is a custom module, not
|
| 217 |
+
# nn.Linear, so quantization leaves the routers in full precision.
|
| 218 |
+
self.weight = mx.zeros((num_experts, dims))
|
| 219 |
+
if bias:
|
| 220 |
+
self.bias = mx.zeros((num_experts,))
|
| 221 |
+
self.top_k = top_k
|
| 222 |
+
self.score_func = score_func
|
| 223 |
+
self.scaling_factor = scaling_factor
|
| 224 |
+
self.renormalize = renormalize
|
| 225 |
+
|
| 226 |
+
def __call__(self, x):
|
| 227 |
+
# Same as the reference: matmul in the model dtype, scores in fp32.
|
| 228 |
+
logits = (x @ self.weight.T).astype(mx.float32)
|
| 229 |
+
if self.score_func == "sigmoid":
|
| 230 |
+
scores = mx.sigmoid(logits)
|
| 231 |
+
elif self.score_func == "softmax":
|
| 232 |
+
scores = mx.softmax(logits, axis=-1, precise=True)
|
| 233 |
+
else:
|
| 234 |
+
raise ValueError(f"Unsupported router score function: {self.score_func}")
|
| 235 |
+
|
| 236 |
+
# The bias is used for selection only.
|
| 237 |
+
selection = scores
|
| 238 |
+
if "bias" in self:
|
| 239 |
+
selection = selection + self.bias.astype(mx.float32)
|
| 240 |
+
|
| 241 |
+
k = self.top_k
|
| 242 |
+
inds = mx.stop_gradient(mx.argpartition(-selection, kth=k - 1, axis=-1)[..., :k])
|
| 243 |
+
# Gather the unbiased scores, as torch.gather(routing_scores, ...) does.
|
| 244 |
+
weights = mx.take_along_axis(scores, inds, axis=-1)
|
| 245 |
+
if self.renormalize:
|
| 246 |
+
weights = weights / weights.sum(axis=-1, keepdims=True)
|
| 247 |
+
return inds, weights * self.scaling_factor
|
| 248 |
+
|
| 249 |
+
|
| 250 |
+
class SwitchValueExperts(nn.Module):
|
| 251 |
+
"""MoVA values: sum_k w_k * silu(x @ V_k) over the selected value experts.
|
| 252 |
+
|
| 253 |
+
Reference: ``combine_routed_experts(..., activation=F.silu)`` called from
|
| 254 |
+
``K2HorizonMoVAAttention.forward``, with ``v_experts`` as an
|
| 255 |
+
``nn.ModuleList`` of 64 bias-free ``nn.Linear`` layers.
|
| 256 |
+
|
| 257 |
+
Differences from the reference:
|
| 258 |
+
- The reference loops over the experts that were hit, runs each one on its
|
| 259 |
+
tokens, and accumulates with ``index_add_``. Here one ``gather_mm`` call
|
| 260 |
+
runs all selected experts. The 64 expert weights are stacked into one
|
| 261 |
+
[64, kv_heads * head_dim, hidden] tensor by ``Model.sanitize``.
|
| 262 |
+
- The weighted sum is done in fp32 and cast back once. The reference
|
| 263 |
+
multiplies and accumulates in bf16.
|
| 264 |
+
|
| 265 |
+
Same as the reference: SiLU is applied to each expert output *before* the
|
| 266 |
+
weighted sum, not after.
|
| 267 |
+
"""
|
| 268 |
+
|
| 269 |
+
def __init__(self, input_dims: int, output_dims: int, num_experts: int):
|
| 270 |
+
super().__init__()
|
| 271 |
+
self.proj = SwitchLinear(input_dims, output_dims, num_experts, bias=False)
|
| 272 |
+
|
| 273 |
+
def __call__(self, x, indices, weights):
|
| 274 |
+
x = mx.expand_dims(x, (-2, -3))
|
| 275 |
+
# Same as SwitchGLU: when there are many (token, expert) pairs, as in
|
| 276 |
+
# prompt processing, sort them by expert so that gather_mm reads each
|
| 277 |
+
# expert's weights in order. For one-token decode steps this is skipped.
|
| 278 |
+
do_sort = indices.size >= 64
|
| 279 |
+
idx = indices
|
| 280 |
+
if do_sort:
|
| 281 |
+
x, idx, inv_order = _gather_sort(x, indices)
|
| 282 |
+
x = nn.silu(self.proj(x, idx, sorted_indices=do_sort))
|
| 283 |
+
if do_sort:
|
| 284 |
+
x = _scatter_unsort(x, inv_order, indices.shape)
|
| 285 |
+
x = x.squeeze(-2)
|
| 286 |
+
return (x * weights[..., None]).sum(axis=-2).astype(x.dtype)
|
| 287 |
+
|
| 288 |
+
|
| 289 |
+
class Attention(nn.Module):
|
| 290 |
+
"""GQA attention with an optional MoVA value path and output gate.
|
| 291 |
+
|
| 292 |
+
Reference: ``K2HorizonAttention`` (dense layers 0-2) and
|
| 293 |
+
``K2HorizonMoVAAttention`` (sparse layers). The two reference classes are
|
| 294 |
+
the same except for the values, so here they are one class:
|
| 295 |
+
- ``mova=False``: values = v_proj(x)
|
| 296 |
+
- ``mova=True``: values come from the routed value experts (``v_router`` +
|
| 297 |
+
``v_experts``). There is no ``v_proj`` in these layers.
|
| 298 |
+
|
| 299 |
+
The values are computed per token from the layer input, so they go into the
|
| 300 |
+
KV cache like normal values. MoVA needs no special cache.
|
| 301 |
+
|
| 302 |
+
Differences from the reference:
|
| 303 |
+
- RoPE: ``mx.fast.rope`` with ``traditional=False``, which is the same
|
| 304 |
+
rotate-half layout as the reference ``apply_rotary_pos_emb``. The
|
| 305 |
+
reference makes cos/sin tables and casts them to the model dtype. The
|
| 306 |
+
fused kernel computes them itself, which can round differently in bf16.
|
| 307 |
+
Positions come from the cache offset, not a ``position_ids`` argument.
|
| 308 |
+
- Attention: the fused ``scaled_dot_product_attention`` kernel with native
|
| 309 |
+
GQA. The reference eager path repeats the KV heads (``repeat_kv``) and
|
| 310 |
+
computes softmax in fp32.
|
| 311 |
+
- Cache and mask: mlx-lm ``KVCache`` and causal mask, not HF ``Cache`` and
|
| 312 |
+
``create_causal_mask``.
|
| 313 |
+
"""
|
| 314 |
+
|
| 315 |
+
def __init__(self, args: ModelArgs, mova: bool):
|
| 316 |
+
super().__init__()
|
| 317 |
+
dim = args.hidden_size
|
| 318 |
+
self.n_heads = n_heads = args.num_attention_heads
|
| 319 |
+
self.n_kv_heads = n_kv_heads = args.num_key_value_heads
|
| 320 |
+
self.head_dim = head_dim = args.head_dim
|
| 321 |
+
self.scale = head_dim**-0.5
|
| 322 |
+
bias = args.attention_bias
|
| 323 |
+
|
| 324 |
+
self.q_proj = nn.Linear(dim, n_heads * head_dim, bias=bias)
|
| 325 |
+
self.k_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=bias)
|
| 326 |
+
self.o_proj = nn.Linear(n_heads * head_dim, dim, bias=bias)
|
| 327 |
+
|
| 328 |
+
self.mova = mova
|
| 329 |
+
if mova:
|
| 330 |
+
self.v_router = Router(
|
| 331 |
+
dim,
|
| 332 |
+
args.mova_num_experts,
|
| 333 |
+
args.mova_num_experts_per_tok,
|
| 334 |
+
args.router_score_func,
|
| 335 |
+
args.router_scaling_factor,
|
| 336 |
+
# calc_router_weights ignores norm_topk_prob. It renormalizes
|
| 337 |
+
# only when top_k > 1. The MoE block is different (see below).
|
| 338 |
+
renormalize=args.mova_num_experts_per_tok > 1,
|
| 339 |
+
bias=args.moe_gate_bias,
|
| 340 |
+
)
|
| 341 |
+
self.v_experts = SwitchValueExperts(
|
| 342 |
+
dim, n_kv_heads * head_dim, args.mova_num_experts
|
| 343 |
+
)
|
| 344 |
+
else:
|
| 345 |
+
self.v_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=bias)
|
| 346 |
+
|
| 347 |
+
self.gate_func = args.attention_gate_func
|
| 348 |
+
if self.gate_func is not None:
|
| 349 |
+
if self.gate_func not in ("silu", "softplus"):
|
| 350 |
+
raise ValueError(f"Unsupported attention gate: {self.gate_func}")
|
| 351 |
+
self.gate_proj = nn.Linear(dim, n_heads * head_dim, bias=False)
|
| 352 |
+
|
| 353 |
+
# Reference: q_norm / k_norm are K2HorizonRMSNorm with one group per
|
| 354 |
+
# head, applied to the full projection before the reshape into heads.
|
| 355 |
+
# This checkpoint has query_key_norm=False.
|
| 356 |
+
self.query_key_norm = args.query_key_norm
|
| 357 |
+
if self.query_key_norm:
|
| 358 |
+
self.q_norm = GroupedRMSNorm(n_heads * head_dim, n_heads, args.rms_norm_eps)
|
| 359 |
+
self.k_norm = GroupedRMSNorm(
|
| 360 |
+
n_kv_heads * head_dim, n_kv_heads, args.rms_norm_eps
|
| 361 |
+
)
|
| 362 |
+
|
| 363 |
+
self.rope = initialize_rope(
|
| 364 |
+
head_dim,
|
| 365 |
+
args.rope_theta,
|
| 366 |
+
traditional=False,
|
| 367 |
+
scaling_config=args.rope_parameters,
|
| 368 |
+
max_position_embeddings=args.max_position_embeddings,
|
| 369 |
+
)
|
| 370 |
+
|
| 371 |
+
def __call__(
|
| 372 |
+
self,
|
| 373 |
+
x: mx.array,
|
| 374 |
+
mask: Optional[mx.array] = None,
|
| 375 |
+
cache: Optional[Any] = None,
|
| 376 |
+
) -> mx.array:
|
| 377 |
+
B, L, D = x.shape
|
| 378 |
+
|
| 379 |
+
queries, keys = self.q_proj(x), self.k_proj(x)
|
| 380 |
+
if self.query_key_norm:
|
| 381 |
+
queries, keys = self.q_norm(queries), self.k_norm(keys)
|
| 382 |
+
# MoVA: route each token to its value experts and mix their outputs.
|
| 383 |
+
if self.mova:
|
| 384 |
+
values = self.v_experts(x, *self.v_router(x))
|
| 385 |
+
else:
|
| 386 |
+
values = self.v_proj(x)
|
| 387 |
+
|
| 388 |
+
queries = queries.reshape(B, L, self.n_heads, -1).transpose(0, 2, 1, 3)
|
| 389 |
+
keys = keys.reshape(B, L, self.n_kv_heads, -1).transpose(0, 2, 1, 3)
|
| 390 |
+
values = values.reshape(B, L, self.n_kv_heads, -1).transpose(0, 2, 1, 3)
|
| 391 |
+
|
| 392 |
+
if cache is not None:
|
| 393 |
+
queries = self.rope(queries, offset=cache.offset)
|
| 394 |
+
keys = self.rope(keys, offset=cache.offset)
|
| 395 |
+
keys, values = cache.update_and_fetch(keys, values)
|
| 396 |
+
else:
|
| 397 |
+
queries = self.rope(queries)
|
| 398 |
+
keys = self.rope(keys)
|
| 399 |
+
|
| 400 |
+
output = scaled_dot_product_attention(
|
| 401 |
+
queries, keys, values, cache=cache, scale=self.scale, mask=mask
|
| 402 |
+
)
|
| 403 |
+
output = output.transpose(0, 2, 1, 3)
|
| 404 |
+
|
| 405 |
+
# Per-head output gate, applied to the attention output before o_proj,
|
| 406 |
+
# same as the reference.
|
| 407 |
+
if self.gate_func is not None:
|
| 408 |
+
gate = self.gate_proj(x).reshape(B, L, self.n_heads, -1)
|
| 409 |
+
if self.gate_func == "silu":
|
| 410 |
+
gate = nn.silu(gate)
|
| 411 |
+
else:
|
| 412 |
+
# Reference: F.softplus(gate, beta=log(2)) = log2(1 + 2^x).
|
| 413 |
+
# Computed in fp32 with logaddexp, which is stable for large x.
|
| 414 |
+
# PyTorch switches to the identity when beta * x > 20; the
|
| 415 |
+
# difference is about e^-20.
|
| 416 |
+
g = gate.astype(mx.float32) * math.log(2)
|
| 417 |
+
gate = (mx.logaddexp(g, 0.0) / math.log(2)).astype(output.dtype)
|
| 418 |
+
output = output * gate
|
| 419 |
+
|
| 420 |
+
return self.o_proj(output.reshape(B, L, -1))
|
| 421 |
+
|
| 422 |
+
|
| 423 |
+
class MLP(nn.Module):
|
| 424 |
+
"""SwiGLU MLP. Same as ``K2HorizonMLP``.
|
| 425 |
+
|
| 426 |
+
Used for the dense layers (``mlp_only_layers``) and for the shared expert.
|
| 427 |
+
"""
|
| 428 |
+
|
| 429 |
+
def __init__(self, dim, hidden_dim):
|
| 430 |
+
super().__init__()
|
| 431 |
+
self.gate_proj = nn.Linear(dim, hidden_dim, bias=False)
|
| 432 |
+
self.down_proj = nn.Linear(hidden_dim, dim, bias=False)
|
| 433 |
+
self.up_proj = nn.Linear(dim, hidden_dim, bias=False)
|
| 434 |
+
|
| 435 |
+
def __call__(self, x) -> mx.array:
|
| 436 |
+
return self.down_proj(swiglu(self.gate_proj(x), self.up_proj(x)))
|
| 437 |
+
|
| 438 |
+
|
| 439 |
+
class SparseMoeBlock(nn.Module):
|
| 440 |
+
"""Routed SwiGLU experts plus optional shared experts.
|
| 441 |
+
|
| 442 |
+
Reference: ``K2HorizonSparseMoeBlock``.
|
| 443 |
+
|
| 444 |
+
Differences from the reference:
|
| 445 |
+
- The routed experts are one ``SwitchGLU`` (gather_mm) instead of an
|
| 446 |
+
``nn.ModuleList`` of ``K2HorizonMLP`` run in a Python loop.
|
| 447 |
+
- The weighted sum of expert outputs is done in fp32. The reference casts
|
| 448 |
+
the routing weights to bf16 and accumulates with ``index_add_`` in bf16.
|
| 449 |
+
- Returns only the hidden states. The reference also returns the router
|
| 450 |
+
logits, which are only used for the training loss.
|
| 451 |
+
"""
|
| 452 |
+
|
| 453 |
+
def __init__(self, args: ModelArgs):
|
| 454 |
+
super().__init__()
|
| 455 |
+
dim = args.hidden_size
|
| 456 |
+
self.gate = Router(
|
| 457 |
+
dim,
|
| 458 |
+
args.num_experts,
|
| 459 |
+
args.num_experts_per_tok,
|
| 460 |
+
args.router_score_func,
|
| 461 |
+
args.router_scaling_factor,
|
| 462 |
+
# The MoE block renormalizes whenever norm_topk_prob is set, also
|
| 463 |
+
# for top-1 (the weight becomes 1.0). MoVA does not (see Attention).
|
| 464 |
+
renormalize=args.norm_topk_prob,
|
| 465 |
+
bias=args.moe_gate_bias,
|
| 466 |
+
)
|
| 467 |
+
self.switch_mlp = SwitchGLU(dim, args.moe_intermediate_size, args.num_experts)
|
| 468 |
+
# Same as the reference: the shared experts are one wider MLP, and its
|
| 469 |
+
# output is added to the routed output without a gate.
|
| 470 |
+
if args.num_shared_experts > 0:
|
| 471 |
+
self.shared_experts = MLP(
|
| 472 |
+
dim, args.moe_intermediate_size * args.num_shared_experts
|
| 473 |
+
)
|
| 474 |
+
|
| 475 |
+
def __call__(self, x: mx.array) -> mx.array:
|
| 476 |
+
inds, scores = self.gate(x)
|
| 477 |
+
y = self.switch_mlp(x, inds)
|
| 478 |
+
y = (y * scores[..., None]).sum(axis=-2).astype(y.dtype)
|
| 479 |
+
if "shared_experts" in self:
|
| 480 |
+
y = y + self.shared_experts(x)
|
| 481 |
+
return y
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
class DecoderLayer(nn.Module):
|
| 485 |
+
"""Pre-norm decoder layer. Same structure as ``K2HorizonDecoderLayer``."""
|
| 486 |
+
|
| 487 |
+
def __init__(self, args: ModelArgs, layer_idx: int):
|
| 488 |
+
super().__init__()
|
| 489 |
+
# Same rule as the reference. A sparse layer uses both the MoE block and
|
| 490 |
+
# MoVA attention; a dense layer uses the dense MLP and v_proj attention.
|
| 491 |
+
# This checkpoint: layers 0-2 are dense, layers 3-47 are sparse.
|
| 492 |
+
is_sparse = (
|
| 493 |
+
layer_idx not in args.mlp_only_layers
|
| 494 |
+
and args.num_experts > 0
|
| 495 |
+
and (layer_idx + 1) % args.decoder_sparse_step == 0
|
| 496 |
+
)
|
| 497 |
+
self.self_attn = Attention(args, mova=is_sparse and args.mova_num_experts > 0)
|
| 498 |
+
if is_sparse:
|
| 499 |
+
self.mlp = SparseMoeBlock(args)
|
| 500 |
+
else:
|
| 501 |
+
self.mlp = MLP(args.hidden_size, args.intermediate_size)
|
| 502 |
+
|
| 503 |
+
self.input_layernorm = GroupedRMSNorm(
|
| 504 |
+
args.hidden_size, args.layernorm_num_groups, args.rms_norm_eps
|
| 505 |
+
)
|
| 506 |
+
self.post_attention_layernorm = GroupedRMSNorm(
|
| 507 |
+
args.hidden_size, args.layernorm_num_groups, args.rms_norm_eps
|
| 508 |
+
)
|
| 509 |
+
|
| 510 |
+
def __call__(
|
| 511 |
+
self,
|
| 512 |
+
x: mx.array,
|
| 513 |
+
mask: Optional[mx.array] = None,
|
| 514 |
+
cache: Optional[Any] = None,
|
| 515 |
+
) -> mx.array:
|
| 516 |
+
h = x + self.self_attn(self.input_layernorm(x), mask, cache)
|
| 517 |
+
return h + self.mlp(self.post_attention_layernorm(h))
|
| 518 |
+
|
| 519 |
+
|
| 520 |
+
class K2HorizonModel(nn.Module):
|
| 521 |
+
"""Embeddings, decoder layers and final norm. Reference: ``K2HorizonModel``.
|
| 522 |
+
|
| 523 |
+
Differences from the reference:
|
| 524 |
+
- The reference computes the RoPE cos/sin once per forward and passes them
|
| 525 |
+
to every layer. Here each attention layer applies RoPE itself, using the
|
| 526 |
+
cache offset as the start position.
|
| 527 |
+
- The mask comes from ``create_attention_mask`` (mlx-lm), not HF
|
| 528 |
+
``create_causal_mask``. Padding in batches is handled by mlx-lm's batched
|
| 529 |
+
generation, not by an ``attention_mask`` argument.
|
| 530 |
+
"""
|
| 531 |
+
|
| 532 |
+
def __init__(self, args: ModelArgs):
|
| 533 |
+
super().__init__()
|
| 534 |
+
self.embed_tokens = nn.Embedding(args.vocab_size, args.hidden_size)
|
| 535 |
+
self.layers = [DecoderLayer(args, i) for i in range(args.num_hidden_layers)]
|
| 536 |
+
self.norm = GroupedRMSNorm(
|
| 537 |
+
args.hidden_size, args.layernorm_num_groups, args.rms_norm_eps
|
| 538 |
+
)
|
| 539 |
+
|
| 540 |
+
def __call__(
|
| 541 |
+
self,
|
| 542 |
+
inputs: mx.array,
|
| 543 |
+
cache=None,
|
| 544 |
+
input_embeddings: Optional[mx.array] = None,
|
| 545 |
+
) -> mx.array:
|
| 546 |
+
if input_embeddings is not None:
|
| 547 |
+
h = input_embeddings
|
| 548 |
+
else:
|
| 549 |
+
h = self.embed_tokens(inputs)
|
| 550 |
+
|
| 551 |
+
if cache is None:
|
| 552 |
+
cache = [None] * len(self.layers)
|
| 553 |
+
|
| 554 |
+
mask = create_attention_mask(h, cache[0])
|
| 555 |
+
|
| 556 |
+
for layer, c in zip(self.layers, cache):
|
| 557 |
+
h = layer(h, mask, c)
|
| 558 |
+
|
| 559 |
+
return self.norm(h)
|
| 560 |
+
|
| 561 |
+
|
| 562 |
+
class Model(nn.Module):
|
| 563 |
+
"""Causal LM head. Reference: ``K2HorizonForCausalLM``.
|
| 564 |
+
|
| 565 |
+
Returns logits only. The reference can also compute the LM loss and the
|
| 566 |
+
load-balancing loss, which are only needed for training.
|
| 567 |
+
"""
|
| 568 |
+
|
| 569 |
+
def __init__(self, args: ModelArgs):
|
| 570 |
+
super().__init__()
|
| 571 |
+
self.args = args
|
| 572 |
+
self.model_type = args.model_type
|
| 573 |
+
self.model = K2HorizonModel(args)
|
| 574 |
+
if not args.tie_word_embeddings:
|
| 575 |
+
self.lm_head = nn.Linear(args.hidden_size, args.vocab_size, bias=False)
|
| 576 |
+
|
| 577 |
+
def __call__(
|
| 578 |
+
self,
|
| 579 |
+
inputs: mx.array,
|
| 580 |
+
cache=None,
|
| 581 |
+
input_embeddings: Optional[mx.array] = None,
|
| 582 |
+
) -> mx.array:
|
| 583 |
+
out = self.model(inputs, cache, input_embeddings)
|
| 584 |
+
if self.args.tie_word_embeddings:
|
| 585 |
+
return self.model.embed_tokens.as_linear(out)
|
| 586 |
+
return self.lm_head(out)
|
| 587 |
+
|
| 588 |
+
def sanitize(self, weights):
|
| 589 |
+
"""Convert the PyTorch checkpoint names and layout to this module.
|
| 590 |
+
|
| 591 |
+
The reference stores every expert as its own nn.Linear. gather_mm needs
|
| 592 |
+
one stacked tensor per projection, so:
|
| 593 |
+
|
| 594 |
+
- ``model.layers.{l}.mlp.experts.{e}.{gate,up,down}_proj.weight``
|
| 595 |
+
(one per expert) -> ``model.layers.{l}.mlp.switch_mlp.{proj}.weight``
|
| 596 |
+
with shape [num_experts, out, in]
|
| 597 |
+
- ``model.layers.{l}.self_attn.v_experts.{e}.weight``
|
| 598 |
+
-> ``model.layers.{l}.self_attn.v_experts.proj.weight``
|
| 599 |
+
with shape [mova_num_experts, out, in]
|
| 600 |
+
|
| 601 |
+
All other names match the reference unchanged. Already converted
|
| 602 |
+
weights pass through, so the function is safe to run twice.
|
| 603 |
+
"""
|
| 604 |
+
if self.args.tie_word_embeddings:
|
| 605 |
+
weights.pop("lm_head.weight", None)
|
| 606 |
+
# RoPE frequencies are computed at run time, not loaded.
|
| 607 |
+
weights = {k: v for k, v in weights.items() if "rotary_emb.inv_freq" not in k}
|
| 608 |
+
|
| 609 |
+
for l in range(self.args.num_hidden_layers):
|
| 610 |
+
prefix = f"model.layers.{l}"
|
| 611 |
+
if f"{prefix}.mlp.experts.0.up_proj.weight" in weights:
|
| 612 |
+
for n in ["up_proj", "down_proj", "gate_proj"]:
|
| 613 |
+
to_join = [
|
| 614 |
+
weights.pop(f"{prefix}.mlp.experts.{e}.{n}.weight")
|
| 615 |
+
for e in range(self.args.num_experts)
|
| 616 |
+
]
|
| 617 |
+
weights[f"{prefix}.mlp.switch_mlp.{n}.weight"] = mx.stack(to_join)
|
| 618 |
+
if f"{prefix}.self_attn.v_experts.0.weight" in weights:
|
| 619 |
+
to_join = [
|
| 620 |
+
weights.pop(f"{prefix}.self_attn.v_experts.{e}.weight")
|
| 621 |
+
for e in range(self.args.mova_num_experts)
|
| 622 |
+
]
|
| 623 |
+
weights[f"{prefix}.self_attn.v_experts.proj.weight"] = mx.stack(to_join)
|
| 624 |
+
return weights
|
| 625 |
+
|
| 626 |
+
@property
|
| 627 |
+
def layers(self):
|
| 628 |
+
return self.model.layers
|
model-00001-of-00005.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d44a0613621053b92a30e7bc5bbbd932ae468e2ae483ba397be8e072e2a7eb5d
|
| 3 |
+
size 5366373156
|
model-00002-of-00005.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c210d48abe8f1d6d4960bc5cae2a9b40ae9f29ba54c1179cf6ef827734be08a1
|
| 3 |
+
size 5364015801
|
model-00003-of-00005.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:928598e2a06d9cb905c016c05f1801cd0addcf548f3f0150f20a0360019645ad
|
| 3 |
+
size 5295256414
|
model-00004-of-00005.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da51bad830ef84aebf2dbf1bf76fea814c8b819f8087daf1e7596e6c572e77f7
|
| 3 |
+
size 5314240687
|
model-00005-of-00005.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:67dfaed3be323a9eab1898abb873ad5b4873f6664f8998261740f04c6dbca551
|
| 3 |
+
size 5059628434
|
model.safetensors.index.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:838d767b7c9925ff257feb20eaa4299a8e3cc35bb3d805589c373f51d2cc3cb6
|
| 3 |
+
size 20637260
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"backend": "tokenizers",
|
| 3 |
+
"bos_token": "<|ifm|begin_of_text|>",
|
| 4 |
+
"clean_up_tokenization_spaces": false,
|
| 5 |
+
"eos_token": "<|ifm|endoftext|>",
|
| 6 |
+
"is_local": true,
|
| 7 |
+
"local_files_only": false,
|
| 8 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 9 |
+
"tokenizer_class": "TokenizersBackend"
|
| 10 |
+
}
|