Dhฤซr 5B-A1.7B ยท 4-bit

Dhฤซr 5B-A1.7B quantised to 4-bit for llama.cpp, for English and Indian languages. At 3.4 GB, it runs on laptops, desktops, mobiles and embedded chips. Because only 1.7B of its parameters work on each token, it generates text faster than Dhฤซr 2B on a CPU.

  • Original model: zenteiq-ai/Dhir-5B-A1.7B
  • Parameters: 5.1B, of which 1.7B are active for each token
  • Context length: 16,384 tokens
  • File: Dhir-5B-A1.7B-Q4_K_M.gguf (3.4 GB)
  • License: Apache 2.0

Choose a version

Version Size Best for
zenteiq-ai/Dhir-2B-4bit 1.3 GB Edge devices, mobiles and embedded chips with little memory
zenteiq-ai/Dhir-2B-8bit 2.2 GB Laptops and mobiles, closest to the original
Prathat/Dhir-5B-A1.7B-4bit 3.4 GB Laptops and desktops; stronger and faster than the 2B on a CPU
zenteiq-ai/Dhir-5B-A1.7B-8bit 5.4 GB Laptops, desktops and small GPUs, closest to the original

Quickstart

Until Dhฤซr support is merged into llama.cpp, build the dhir-support branch:

git clone -b dhir-support https://github.com/zenteiq-research/llama.cpp
cd llama.cpp
cmake -B build            # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build -j --target llama-server llama-cli

Then run the model by its repository name; llama.cpp downloads the file on first use:

# Server with an OpenAI-compatible API on port 8080
./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080

# Chat in the terminal
./build/bin/llama-cli -hf Prathat/Dhir-5B-A1.7B-4bit --jinja

On a CPU, set -t to your number of performance cores and -tb to your total number of cores for the best speed, for example -t 8 -tb 24.

Python (LangChain)

pip install -U langchain-openai
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8080/v1",
    api_key="EMPTY",
    model="Prathat/Dhir-5B-A1.7B-4bit",
    temperature=0.27,
    top_p=0.98,
    presence_penalty=0.25,
    frequency_penalty=0,
    extra_body={"min_p": 0.06, "repeat_penalty": 1.025},
)
print(llm.invoke("เคญเคพเคฐเคค เค•เฅ€ เคฐเคพเคœเคงเคพเคจเฅ€ เค•เฅ‡ เคฌเคพเคฐเฅ‡ เคฎเฅ‡เค‚ เคฆเฅ‹ เคตเคพเค•เฅเคฏ เคฒเคฟเค–เคฟเคเฅค").content)

Memory

The model needs about 3.4 GB plus 0.07 MB for each token of conversation. For long conversations on devices with little memory, an 8-bit context cache halves the second part:

./build/bin/llama-server -hf Prathat/Dhir-5B-A1.7B-4bit --jinja --port 8080 -fa on -ctk q8_0 -ctv q8_0

Quality

We tested this file against the original model on WikiText-2 perplexity, on the KL divergence of its next-token predictions over 1.5 million tokens in 11 languages, code and mathematics, and on standard benchmarks. Its perplexity is within 1.8% of the original, and it picks the same next token 91% of the time.

For the full evaluation, see our technical paper and zenteiq.ai/research/blog.

Limitations

  • 4-bit quantisation gives up a little accuracy for its size; use the 8-bit version when memory allows.
  • For multi-step arithmetic in Bengali and Telugu, prefer the 8-bit version.
  • Like any language model, Dhฤซr can state incorrect facts fluently. Verify what matters, and add your own safeguards in production.

Files

File Size SHA-256
Dhir-5B-A1.7B-Q4_K_M.gguf 3.4 GB 6e137eb3d1713419c73c9a4cad1ea6e851f6f9aa9c361c4715492c62cda5e1c9

License

Apache 2.0, the same as the original model. The LICENSE and NOTICE files of zenteiq-ai/Dhir-5B-A1.7B are included in this repository.

Downloads last month
240
GGUF
Model size
5B params
Architecture
dhirmoe
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Prathat/Dhir-5B-A1.7B-GGUF

Quantized
(3)
this model