Instructions to use swarmagents/Qwen3.8-27B-Direct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Use Docker
docker model run hf.co/swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "swarmagents/Qwen3.8-27B-Direct-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "swarmagents/Qwen3.8-27B-Direct-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
- Ollama
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with Ollama:
ollama run hf.co/swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with Docker Model Runner:
docker model run hf.co/swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
- Lemonade
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwen3.8-27B-Direct-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use swarmagents/Qwen3.8-27B-Direct-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-27B-Direct (GGUF)
In-Distribution Contrastive Orthogonal Weight Steering on Qwen/Qwen3.8-27B
Zero Circular Rambling. Surgical Reasoning. Full Multi-Token Prediction (MTP) Speculative Alignment.
๐ก What is Qwen3.8-27B-Direct?
Base modern reasoning models frequently suffer from circular self-doubt in <think> sequences: repeating phrases like "Wait, let me double check this again..." 8 to 12 times before finally arriving at the answer. This inflates context windows, wastes user time, and frequently causes the model to second-guess its own correct deductions.
Qwen3.8-27B-Direct resolves this by mathematically isolating and dampening the rambling subspace directly inside the model's weights using In-Distribution Contrastive Orthogonal Weight Steering.
Highlights
- โก Shorter Thinking: Drops mean thinking tokens from
420 down to **85 per query** (-79.8%), about 1.8ร faster time-to-solution on a local M4 Max (internal eval). - ๐ฏ Higher Accuracy on Math & Code: +0.7% on GSM8K (89.2%) and +1.2% on HumanEval (78.6%) by preventing circular hallucination loops.
- ๐ SVD Orthogonalization against Answer Subspace: Steering vectors are strictly orthogonalized against the top-8 principal components of post-
</think>answer tokens infloat32, ensuring core reasoning capabilities remain untouched. - ๐ก๏ธ Frobenius Norm Restoration: Every steered tensor's energy is scaled to match its original Frobenius norm: $$W_{\text{new}} = W_{\text{proj}} \cdot \frac{|W|F}{|W{\text{proj}}|_F + 10^{-8}}$$
- ๐ Full MTP Speculative Decoding Preservation: Created via direct streaming safetensors surgery (bypassing Hugging Face
AutoModel.save_pretrained()which silently dropsmtp.*weights). All 15 MTP weights and 333 vision weights are 100% intact, enabling +70% speculative decoding throughput inllama.cppandvLLM.
๐ Benchmark Results
| Benchmark | Base Qwen 3.8 27B | Qwen3.8-27B-Direct | Delta |
|---|---|---|---|
| Hermes Bench (General Capability) | 65.77% | 65.60% | ~0.0% (Zero Forgetting) |
| Terminal-Bench-2.1 (CLI & Systems) | 67.00% | 70.00% | +3.0% Gain |
| GSM8K (8-shot CoT) | 88.50% | 89.20% | +0.7% Gain |
| HumanEval (0-shot Python) | 77.40% | 78.60% | +1.2% Gain |
| Coffman Concurrency & Logic | 100% | 100% | 100% Retained |
| Mean Thinking Token Count | ~420 tokens | ~85 tokens | -79.8% (~4.9ร fewer tokens) |
| Time to Solution (Local M4 Max) | 65.6s | 36.2s | 1.8ร faster turnaround |
| Multi-Token Prediction (MTP) Speedup | +70% | +70% | Full Alignment Maintained |
๐ ๏ธ Architectural Details
- Base Architecture:
Qwen3_5ForConditionalGeneration(model_type: "qwen3_5"). - Steering Target Layers: Full Gated Attention layers
[23, 27, 31, 35, 39, 43, 47, 51]. - Target Projections:
self_attn.o_proj.weight($[5120, 6144]$) andmlp.down_proj.weight($[5120, 17408]$). - Projection Parameter: Rank-1 damped orthogonal projection with $\beta = 0.25$.
- Steering Formula: $$P = \hat{v}_l \cdot \hat{v}l^T \quad \text{where } \hat{v}l \perp U{\text{cap}}^{(l)}$$ $$W{\text{proj}} = W - \beta (P W)$$
๐ Quickstart
Option 1: 1-Click with Ollama
# Easiest: pull straight from Hugging Face
ollama run hf.co/swarmagents/Qwen3.8-27B-Direct-GGUF:Q4_K_M
Or import locally with the bundled Modelfile:
# Clone the repository
git clone https://huggingface.co/swarmagents/Qwen3.8-27B-Direct-GGUF
cd Qwen3.8-27B-Direct-GGUF
# Create and run in Ollama
ollama create qwen3.8-27b-direct -f Modelfile
ollama run qwen3.8-27b-direct "Implement an asynchronous lock-free priority queue in Rust."
Option 2: llama.cpp with Native Speculative Decoding (MTP)
Because all 15 mtp.* weights were preserved, you can enable native speculative drafting:
./llama-cli \
-m Qwen3.8-27B-Direct-Q4_K_M.gguf \
--draft-max 2 \
-p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nSolve the 2D heat equation with mixed boundary conditions.<|im_end|>\n<|im_start|>assistant\n<think>\n"
๐ฆ Provided Files
| File | Size | Description | Recommended Hardware |
|---|---|---|---|
Qwen3.8-27B-Direct-Q4_K_M.gguf |
16.8 GB (15.7 GiB) | Highly optimized 4-bit Medium K-quant | Apple Silicon (24GB+ unified RAM), RTX 3090 / 4090 (24GB) |
Modelfile |
1 KB | Pre-configured Ollama import file with official chat template | Any |
๐ค Citation & Credits
Built by SwarmAgents on Apple Silicon (M4 Max) using MLX and llama.cpp.
@misc{qwen38_27b_direct_2026,
author = {SwarmAgents},
title = {Qwen3.8-27B-Direct: In-Distribution Contrastive Orthogonal Weight Steering for Zero-Rambling Reasoning},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/swarmagents/Qwen3.8-27B-Direct-GGUF}}
}
- Downloads last month
- 135
4-bit
Model tree for swarmagents/Qwen3.8-27B-Direct-GGUF
Base model
Qwen/Qwen3.8-27B