Instructions to use Prathat/Dhir-5B-A1.7B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Prathat/Dhir-5B-A1.7B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use Prathat/Dhir-5B-A1.7B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Prathat/Dhir-5B-A1.7B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Prathat/Dhir-5B-A1.7B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
- Ollama
How to use Prathat/Dhir-5B-A1.7B-GGUF with Ollama:
ollama run hf.co/Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use Prathat/Dhir-5B-A1.7B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Prathat/Dhir-5B-A1.7B-GGUF with Docker Model Runner:
docker model run hf.co/Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
- Lemonade
How to use Prathat/Dhir-5B-A1.7B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Dhir-5B-A1.7B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use Prathat/Dhir-5B-A1.7B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Prathat/Dhir-5B-A1.7B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Prathat/Dhir-5B-A1.7B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Dhฤซr 5B-A1.7B ยท 4-bit
Dhฤซr 5B-A1.7B quantised to 4-bit for llama.cpp, for English and Indian languages. At 3.4 GB, it runs on laptops, desktops, mobiles and embedded chips. Because only 1.7B of its parameters work on each token, it generates text faster than Dhฤซr 2B on a CPU.
- Original model: zenteiq-ai/Dhir-5B-A1.7B
- Parameters: 5.1B, of which 1.7B are active for each token
- Context length: 16,384 tokens
- File:
Dhir-5B-A1.7B-Q4_K_M.gguf(3.4 GB) - License: Apache 2.0
Choose a version
| Version | Size | Best for |
|---|---|---|
| zenteiq-ai/Dhir-2B-4bit | 1.3 GB | Edge devices, mobiles and embedded chips with little memory |
| zenteiq-ai/Dhir-2B-8bit | 2.2 GB | Laptops and mobiles, closest to the original |
| Prathat/Dhir-5B-A1.7B-4bit | 3.4 GB | Laptops and desktops; stronger and faster than the 2B on a CPU |
| zenteiq-ai/Dhir-5B-A1.7B-8bit | 5.4 GB | Laptops, desktops and small GPUs, closest to the original |
Quickstart
Until Dhฤซr support is merged into llama.cpp, build the dhir-support branch:
git clone -b dhir-support https://github.com/zenteiq-research/llama.cpp
cd llama.cpp
cmake -B build # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build -j --target llama-server llama-cli
Then run the model by its repository name; llama.cpp downloads the file on first use:
# Server with an OpenAI-compatible API on port 8080
./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080
# Chat in the terminal
./build/bin/llama-cli -hf Prathat/Dhir-5B-A1.7B-4bit --jinja
On a CPU, set -t to your number of performance cores and -tb to your total number of cores for the best speed, for example -t 8 -tb 24.
Python (LangChain)
pip install -U langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="http://localhost:8080/v1",
api_key="EMPTY",
model="Prathat/Dhir-5B-A1.7B-4bit",
temperature=0.27,
top_p=0.98,
presence_penalty=0.25,
frequency_penalty=0,
extra_body={"min_p": 0.06, "repeat_penalty": 1.025},
)
print(llm.invoke("เคญเคพเคฐเคค เคเฅ เคฐเคพเคเคงเคพเคจเฅ เคเฅ เคฌเคพเคฐเฅ เคฎเฅเค เคฆเฅ เคตเคพเคเฅเคฏ เคฒเคฟเคเคฟเคเฅค").content)
Memory
The model needs about 3.4 GB plus 0.07 MB for each token of conversation. For long conversations on devices with little memory, an 8-bit context cache halves the second part:
./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080 -fa on -ctk q8_0 -ctv q8_0
Quality
We tested this file against the original model on WikiText-2 perplexity, on the KL divergence of its next-token predictions over 1.5 million tokens in 11 languages, code and mathematics, and on standard benchmarks. Its perplexity is within 1.8% of the original, and it picks the same next token 91% of the time.
For the full evaluation, see our technical paper and zenteiq.ai/research/blog.
Limitations
- 4-bit quantisation gives up a little accuracy for its size; use the 8-bit version when memory allows.
- For multi-step arithmetic in Bengali and Telugu, prefer the 8-bit version.
- Like any language model, Dhฤซr can state incorrect facts fluently. Verify what matters, and add your own safeguards in production.
Files
| File | Size | SHA-256 |
|---|---|---|
Dhir-5B-A1.7B-Q4_K_M.gguf |
3.4 GB | 6e137eb3d1713419c73c9a4cad1ea6e851f6f9aa9c361c4715492c62cda5e1c9 |
License
Apache 2.0, the same as the original model. The LICENSE and NOTICE files of zenteiq-ai/Dhir-5B-A1.7B are included in this repository.
- Downloads last month
- 240
4-bit
Model tree for Prathat/Dhir-5B-A1.7B-GGUF
Base model
zenteiq-ai/Dhir-5B-A1.7B