Instructions to use AngelSlim/Hy4-flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AngelSlim/Hy4-flash-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Use Docker
docker model run hf.co/AngelSlim/Hy4-flash-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use AngelSlim/Hy4-flash-GGUF with Ollama:
ollama run hf.co/AngelSlim/Hy4-flash-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use AngelSlim/Hy4-flash-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AngelSlim/Hy4-flash-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AngelSlim/Hy4-flash-GGUF with Docker Model Runner:
docker model run hf.co/AngelSlim/Hy4-flash-GGUF:Q4_K_M
- Lemonade
How to use AngelSlim/Hy4-flash-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AngelSlim/Hy4-flash-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Hy4-flash-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use AngelSlim/Hy4-flash-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AngelSlim/Hy4-flash-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AngelSlim/Hy4-flash-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AngelSlim/Hy4-flash-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Download README_CN.md from AngelSlim/Hy4-flash-GGUF: direct link, hf CLI and curl.
- Browser
- Download file 4.62 kB
-
https://huggingface.co/AngelSlim/Hy4-flash-GGUF/resolve/main/README_CN.md
- Command line
-
hf download hf://AngelSlim/Hy4-flash-GGUF/README_CN.md
-
curl -L -o README_CN.md https://huggingface.co/AngelSlim/Hy4-flash-GGUF/resolve/main/README_CN.md
English | 中文
A more accessible, comprehensive, and efficient toolkit for large model compression.
✒️ TechnicalReport | 📖 Documentation | 🤗 Hugging Face | 🤖 ModelScope
Hy4-flash GGUF
Hy4-flash 的五个 GGUF 版本:tencent/Hy4-flash
| 文件 | 模型大小 | bpw(整体) | bpw(不含 Engram 表) |
|---|---|---|---|
Hy4-flash-Q4_K_M.gguf |
156.79 GiB | 6.05 | 5.27 |
Hy4-flash-SPARK-4Bit.gguf |
151.3 GiB | 5.84 | 4.78 |
Hy4-flash-SPARK-4Bit-Fast.gguf |
148.6 GiB | 5.74 | 4.63 |
Hy4-flash-Q3_K.gguf |
144.70 GiB | 5.58 | 4.62 |
Hy4-flash-IQ2-AH.gguf |
125.73 GiB | 4.85 | 3.60 |
1. 这是什么
Hy4-flash-Q4_K_M.gguf —— 标准 4 bit 版本。路由专家沿用 llama.cpp 的 Q4_K_M 规则:gate/up 为
Q4_K,down 按 llama.cpp 的固定规则在 19 层用 Q6_K、其余 22 层用 Q4_K。多数非专家权重为 Q8_0。没有显存压力就用这个。
Hy4-flash-SPARK-4Bit.gguf —— SPARK-4Bit 版本(NVIDIA 的 4 bit 浮点格式)。适用于 Spark 设备。
Hy4-flash-SPARK-4Bit-Fast.gguf —— SPARK-4Bit 的变体,比 SPARK-4Bit 小 2.7 GiB。同样适用于 Spark 设备。
Hy4-flash-Q3_K.gguf —— 专家层 Q3_K/Q4_K 混合。路由专家的 gate/up 在按量化敏感性选出的
19 层保留 Q4_K,其余 22 层用 Q3_K;down 为 Q4_K。路由专家以外的张量与 Q4_K_M 相同。
Hy4-flash-IQ2-AH.gguf —— 五个版本中体积最小。路由专家的 gate/up 用 2.0625 bpw(IQ2_XXS),down 用 Q4_K;多数非专家权重保留 BF16,而不是 Q8_0。
2. SPEED
测试于真实workbuddy场景和数据。
Mac M5 Max 128GB
| 格式 | Prefill tok/s (4K / 16K / 32K) | Decode tok/s (4K / 16K / 32K) |
|---|---|---|
| Q4_K_M | 629.8 / 448.1 / 418.6 | 48.13 / 45.18 / 44.15 |
| Q3_K | 743.2 / 563.0 / 520.6 | 50.34 / 49.47 / 47.37 |
对于 code 生成场景,Q3_K 可达 70 tok/s。
RTX Spark 128GB
| format | Prefill tok/s (4K / 16K / 32K) | Decode tok/s (4K / 16K / 32K) |
|---|---|---|
| SPARK-4Bit-Fast | 799 / 717 / 707 | 31.95 / 33.14 / 33.39 |
| SPARK-4Bit | 765 / 686 / 666 | 25.80 / 28.16 / 30.3 |
对于 code 生成场景,SPARK-4Bit-Fast 可达 40 tok/s。
3. 如何使用
通用运行 相关支持在 PR #32517 中,还没有合并,需要拉取 PR 分支:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/32517/head:hy4-flash
git checkout hy4-flash
编译
cmake -B build # NVIDIA 显卡加 -DGGML_CUDA=ON;macOS 默认启用 Metal
cmake --build build --config Release -j
终端对话:
./build/bin/llama-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja
OpenAI 兼容服务(http://localhost:8080):
./build/bin/llama-server -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja --port 8080
图片输入(需要 mmproj 文件):
./build/bin/llama-mtmd-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-Hy4-flash-bf16.gguf \
--image photo.png -p "描述这张图片。" -c 8192 -ngl 99 --jinja
Mac M5 Max 128GB 优化版本
RTX Spark 128GB 优化版本
