SpearsoftLabs/spearindic-tokenizer

SpearIndic Tokenizer v2

A lossless SentencePiece Unigram tokenizer for the 22 scheduled Indian languages + English, trained on three writing styles: native script (all 22 languages), and romanized (Latin letters) and code-mixed (Indic + English) text for the 11 languages that have such training data (as, bn, gu, hi, kn, ml, mr, or, pa, ta, te).

Status: research release. Tokenizer only; no language model has been trained with it yet. Earlier v1 files were removed from this repository; v1 numbers are no longer valid.

Paper: SpearIndic: A Lossless Tokenizer for Native-Script, Romanized and Code-Mixed Text in Indian Languages (preprint, Zenodo, 2026).

Highlights

• Beats IndicBERTv2 overall on native script and code-mixed text with a 20% smaller vocabulary: +4.7% bytes per token on IN22-Gen, +1.0% on FLORES-200, +2.0% on code-mixed text.
• Lossless: 100% exact round-trip on 97,750 test texts (IndicBERTv2: 0.75%).
• Most compact lossless tokenizer we tested on every Indic test set; less than half the tokens of GPT-4o's tokenizer on native script.
• One vocabulary, three styles: native script for 22 languages; romanized and code-mixed text for 11 of them.

Style	Example (Telugu)	Tokens
Native	నేను రేపు హైదరాబాద్ వెళ్తున్నాను.	6
Romanized	nenu repu Hyderabad veltunnanu.	9
Code-mixed	నేను రేపు office కి వెళ్తున్నాను, meeting ఉంది.	10

Files

Location	Model	Vocab
repo root	Unigram-200k (default)	200,000
unigram-128k/	Unigram-128k (smaller)	128,000
ablations/	14 research models from the paper (SentencePiece files only)	16,000–128,000

Each folder has the Hugging Face files (tokenizer.json, …) and the plain SentencePiece file spearindic.model. Both give the same token ids on all but 3 (200k) and 5 (128k) of 162,630 test texts. Those few are exact ties (same number of pieces, same score, different order); decoding is identical on every text.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("SpearsoftLabs/spearindic-tokenizer")
ids = tok.encode("నేను రేపు office కి వెళ్తున్నాను", add_special_tokens=False)
print(tok.convert_ids_to_tokens(ids)); print(tok.decode(ids))

Behaviour

• Lossless: no normalization inside the tokenizer, and byte fallback for unseen characters, so decode(encode(x)) == x. Measured: 100% exact on all 97,750 test texts. Spaces are kept exactly.
• Full script coverage: every letter of the 12 relevant Unicode script blocks has its own piece, including precomposed letters such as য় ਸ਼ ड़, so they never fall back to bytes.
• Optional NFC: training text was NFC-normalized. Calling unicodedata.normalize("NFC", text) first saves some tokens on Bengali, Punjabi and Assamese text, but decode then returns the NFC text, not your original bytes.
• Special tokens: 0 <unk>, 1 <s>, 2 </s>, 3 <pad>. Nothing is added automatically.

Benchmark

Frozen test set chosen before training (97,750 texts; no test text exactly matches a training line). Bytes per token (higher = fewer tokens), and exact round-trip over all texts.

Tokenizer	Vocab	IN22-Gen native	FLORES-200 native	Romanized ²	Code-mixed ²	English	Round-trip
SpearIndic-Unigram-200k	200,000	9.28	9.56	5.17	10.14	4.58	100.00%
IndicBERTv2 ¹	250,000	8.86	9.47	4.21	9.94	4.88	0.75%
SUTRA	256,061	7.32	7.93	4.31	9.11	5.43	81.94%
XLM-R	250,002	7.22	7.38	3.83	7.61	4.20	78.36%
GPT-4o o200k_base	200,019	3.99	4.54	4.02	6.55	4.76	100.00%
GPT-4 cl100k_base	100,277	1.82	1.81	3.70	2.17	4.68	100.00%

¹ IndicBERTv2's round-trip is near 0% as released because its tokenizer file has no decoder ("##" markers stay in the output). With the standard WordPiece decoder added it reaches 27% on FLORES-200 Indic text; the rest fail because it discards spaces around punctuation. Its token counts are not affected. ² Held-out sentences from IndicCMix, the same dataset used for training; no baseline saw it, so these numbers favour SpearIndic.

Read this before quoting the table:

• On native script, SpearIndic needs fewer tokens than IndicBERTv2 in 10 of 22 IN22-Gen languages. The overall lead comes from low-resource languages (Manipuri, Santali, Bodo, Kashmiri). IndicBERTv2 is still better on the 11 larger languages (Hindi, Bengali, Tamil, Telugu, …) by 1–20%.
• Real romanized text: on Dakshina (human romanizations, never used in training) the lead over IndicBERTv2 disappears (table below; ahead in 4 of 11 languages). The romanized column above is in-distribution.
• English is weaker than GPT tokenizers (4.58 vs 4.76 bytes per token).
• Token premium vs English on IN22-Gen (fixed o200k English reference): SpearIndic 1.33, IndicBERTv2 1.40, o200k 3.10.

Out-of-distribution romanized text (Dakshina test, 54,880 sentences, 11 languages)

Tokenizer	Bytes per token	Round-trip
SpearIndic-Unigram-200k	3.29	100.00%
IndicBERTv2 ¹	3.33	0.98%
SUTRA	3.15	99.86%
XLM-R	3.06	99.79%
GPT-4o o200k_base	2.99	100.00%
GPT-4 cl100k_base	2.67	100.00%

Ablation models (research only)

The folder ablations/ holds the 14 models of the paper's vocabulary/language ladder (5, 11 and 22 languages; Unigram and BPE). They let anyone reproduce the paper's ablation tables. Do not use them in products: they were trained before the full-script-coverage fix, so some precomposed letters (for example য় ਸ਼ ड़) fall back to bytes. Load them with sentencepiece:

import sentencepiece as spm
from huggingface_hub import hf_hub_download
sp = spm.SentencePieceProcessor(model_file=hf_hub_download("SpearsoftLabs/spearindic-tokenizer", "ablations/22lang-bpe-128k.model"))

There is no BPE model with the fix: SentencePiece's BPE trainer stops when forced letters never occur in the training text.

File	Languages	Algorithm	Vocab	Bytes/token, IN22-Gen (5 core languages)	Bytes/token, English
ablations/11lang-bpe-128k.model	11	BPE	128k	10.58	4.48
ablations/11lang-bpe-64k.model	11	BPE	64k	9.26	4.13
ablations/11lang-unigram-128k.model	11	Unigram	128k	10.83	4.43
ablations/11lang-unigram-64k.model	11	Unigram	64k	9.50	4.09
ablations/22lang-bpe-128k.model	22	BPE	128k	10.14	4.42
ablations/22lang-bpe-64k.model	22	BPE	64k	8.86	4.05
ablations/22lang-unigram-128k.model	22	Unigram	128k	10.37	4.38
ablations/22lang-unigram-64k.model	22	Unigram	64k	9.07	4.03
ablations/5lang-bpe-16k.model	5	BPE	16k	7.83	3.33
ablations/5lang-bpe-32k.model	5	BPE	32k	9.08	3.78
ablations/5lang-bpe-64k.model	5	BPE	64k	10.42	4.18
ablations/5lang-unigram-16k.model	5	Unigram	16k	7.99	3.30
ablations/5lang-unigram-32k.model	5	Unigram	32k	9.29	3.73
ablations/5lang-unigram-64k.model	5	Unigram	64k	10.61	4.13

Training data

Style	Source	Per language
Native script	ai4bharat/IndicCorpV2	up to 50,000 lines (22 languages)
Romanized	ai4bharat/IndicCMix romanized_casual	30,000 rows (11 languages)
Code-mixed	ai4bharat/IndicCMix native_script_codemixed	30,000 rows (11 languages)
English	IndicCMix English + UltraChat 200k	15% of all characters

2.1M lines, 371M characters. SentencePiece settings: model_type=unigram, byte_fallback=True, character_coverage=0.9995 plus required_chars (full script blocks), normalization_rule_name=identity, remove_extra_whitespaces=False, input_sentence_size=0.

Limitations

• Romanized and code-mixed training text comes only from IndicCMix, whose generation method is not documented. On real human romanizations (Dakshina) SpearIndic is only on par with IndicBERTv2.
• Very little training text for Dogri, Kashmiri and Manipuri.
• Token counts only: fewer tokens do not by themselves mean a better language model.

Citation

@misc{rachamalla2026spearindic,
  title     = {SpearIndic: A Lossless Tokenizer for Native-Script, Romanized and Code-Mixed Text in Indian Languages},
  author    = {Rachamalla, Srish and Ahmad, MD Waseem and Goli, Srinivas and Jaiswal, Naman and Elluru, Harshitha and Palvai, Ajay},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23665757},
  url       = {https://doi.org/10.5281/zenodo.23665757}
}

License

Tokenizer files: Apache-2.0. Training data: IndicCorpV2 (CC0), IndicCMix (MIT, gated), UltraChat 200k (MIT; generated with ChatGPT). Evaluation only: FLORES-200 (CC BY-SA 4.0), IN22-Gen (CC BY 4.0).

Developed by Spearsoft (spearsoft.ai).