arunp0/chitragupta-pii-indic

Chitragupta: PII detection and redaction for Indian languages and Hinglish (Aadhaar, PAN, UPI, phone, names)

Chitragupta (chitragupta-pii-indic) is an open-weights, Apache-2.0 model that finds personal data in Hindi, Hinglish, 10 other Indian languages and English. It tags 18 types of personal data, including Indian identifiers such as Aadhaar, PAN, voter ID, GSTIN, UPI ID and IFSC, and returns character offsets you can mask. It is a 560M-parameter token tagger (XLM-RoBERTa-large) plus a small rule layer that uses checksums and formats to fix ID labels.

On a fresh test set written by a different model, with ID formats never seen in training (1,582 texts, 13 languages)	Exact-span F1
Chitragupta, model + rules	0.959
Chitragupta, model only	0.876

Exact-span F1 counts a prediction as correct only when the label and both character boundaries match. For redaction, where only the masked characters matter, see Comparison with other PII models. All numbers on this card come from synthetic test data; read Evaluation and Limitations before relying on them.

Key facts

• What: a named-entity recognition (NER) model for personal data (PII) in Indian text, with a Python detect() / redact() API that returns character offsets.
• Languages: Hindi, Hinglish (romanised Hindi mixed with English), Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and Indian English, in native scripts and romanised.
• Finds: Aadhaar, PAN, voter ID (EPIC), passport, driving licence, vehicle registration, UPI ID, bank account, IFSC, card number, GSTIN, phone, email, address, pincode, date of birth, person names and IP addresses.
• Runs: locally on CPU or GPU, no API calls; Apache-2.0, free for commercial use.

Use it for: masking personal data before text is logged, stored, shared with an LLM or used for analytics, in support chats, call-centre transcripts, emails, forms, SMS and social posts written in Hindi (Devanagari or romanised), Hinglish, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and Indian English.

The name. In Indian tradition Chitragupta is the keeper of records, who writes down the details of every person's life. This model looks for those details so they can be protected. It is a companion to Yama, a zero-shot classifier for the same languages.

Chitragupta is an independent personal release.

Parameters	about 560M (XLM-RoBERTa-large encoder plus a token-classification head, 37 BIO tags)
Input	any length; texts over 510 tokens are split into overlapping 256-token windows (accuracy is lower on long inputs, see Limitations)
Labels	18 (below)
Training	3 epochs over 114,431 synthetic texts in 13 languages (13,514 with no personal data)
Speed	about 24 ms per short text on a multi-core server CPU in batches of 32; much faster on GPU
Licence	Apache-2.0

Labels

Group	Labels
People	PERSON, DOB
Contact	PHONE, EMAIL, ADDRESS, PINCODE
Government IDs	AADHAAR, PAN, VOTER_ID, PASSPORT, DRIVING_LICENCE, VEHICLE_REG
Financial	UPI_ID, BANK_ACCOUNT, IFSC, CARD_NUMBER, GSTIN
Technical	IP_ADDRESS

Values are found in Latin and native scripts, with native digits (९८७६…, ৯৮৭৬…), with spaces, dashes or dots between groups, partly masked (XXXX XXXX 1234) and, in transcripts, spoken out as words.

Quickstart

# pip install torch transformers huggingface_hub
import sys
from huggingface_hub import snapshot_download

path = snapshot_download("arunp0/chitragupta-pii-indic")
sys.path.insert(0, path)
from chitragupta import Chitragupta

m = Chitragupta(path)                       # GPU if available
text = ("Hi, main Rohit Verma bol raha hoon. Mera Aadhaar XXXX XXXX 5246 hai aur UPI rohit.example@upi, "
        "refund isi pe bhej do. Call: 98765 43210")
m.detect(text)
# [{'label': 'PERSON', 'start': 9, 'end': 20, 'text': 'Rohit Verma'},
#  {'label': 'AADHAAR', 'start': 49, 'end': 63, 'text': 'XXXX XXXX 5246'},
#  {'label': 'UPI_ID', 'start': 76, 'end': 93, 'text': 'rohit.example@upi'},
#  {'label': 'PHONE', 'start': 124, 'end': 135, 'text': '98765 43210'}]
m.redact(text)
# 'Hi, main [PERSON] bol raha hoon. Mera Aadhaar [AADHAAR] hai aur UPI [UPI_ID], refund isi pe bhej do. Call: [PHONE]'

m.detect("Order #4410293 delivered on 12/03/2025 at 10:30. Thanks!")
# []
m.detect_batch(list_of_texts)               # batched; rules=False returns the model's raw spans

The example values are dummies: a masked Aadhaar (the model also finds masked IDs), the widely used placeholder phone number 98765 43210 and the standard sample PAN ABCDE1234F. chitragupta.py is the supported way to run the model: it decodes the tags the way the evaluation did, windows long texts and applies the rule layer (chitragupta_rules.py, which uses the validators in chitragupta_ids.py). It needs Python 3.10+, torch and transformers. Loading the weights with the stock transformers token-classification pipeline also works, and the inference widget on this page uses it, but that route skips the rules, so expect results closer to the model-only row above.

How it works

1. Tagger. XLM-RoBERTa-large fine-tuned to tag each token with one of 37 BIO tags. Spans are decoded to character offsets.

2. Rules (chitragupta_rules.py), applied to the tagger's spans:

   • glue ID fragments back together when the tagger splits an unfamiliar format into pieces with mixed labels;
   • widen a span by one neighbouring digit group when that makes a checksum pass;
   • snap pieces of an IPv4 or IPv6 address to the whole address;
   • relabel a span when exactly one ID type's checksum or format accepts it (Verhoeff for Aadhaar, Luhn for cards, the GSTIN check character, and the PAN, IFSC, voter ID, passport, vehicle and phone formats).

   The rules never change PERSON or ADDRESS spans, and never relabel BANK_ACCOUNT or DRIVING_LICENCE, which have no checksum. They change labels much more than coverage: character-level redaction F1 is 0.984 without them and 0.987 with them.

Training data

All training text is synthetic. No real personal data, production data or public PII dataset was used.

• Text. Gemma 4 26B-A4B-it (open weights, Apache-2.0; the unsloth/gemma-4-26B-A4B-it copy of Google's release) wrote 114,431 texts across 14 document types (support chats, WhatsApp chats, call-centre transcripts, KYC forms, emails, bank SMS, loan notes, courier instructions, HR onboarding, clinic messages, police complaints, rental agreements, social posts and scam messages) in 13 languages: Hinglish 20,546, Hindi 16,331, English 15,080, and 5,178–7,914 each for Bengali, Marathi, Telugu, Tamil, Odia, Punjabi, Gujarati, Urdu, Kannada and Malayalam. 13,514 texts contain no personal data.
• ID and contact labels come from code, not from an LLM. Gemma wrote placeholders such as {AADHAAR_1}; code filled them with generated values (checksum-valid where the ID has a checksum, in many renderings), so those span offsets are exact. Names were filled the same way from a name list generated by Gemma. Addresses are the exception: Gemma wrote them and marked them with tags, so address boundaries are where Gemma put the tags.
• Cross-check. Qwen3.6 (Qwen/Qwen3.6-35B-A3B, open weights, Apache-2.0) read every text and flagged personal data that was not labelled; flagged texts were dropped.
• No output of a proprietary model is in the training data.

Evaluation

Test set

The test set was built to make memorising the training distribution not enough:

• A different writer. Texts were written by Qwen3.6-35B-A3B (a 4-bit AWQ build), not Gemma.
• Unseen document types. 10 document types that never appear in training (matrimonial profiles, admission forms, court notices, sale deeds, voice-note transcripts, cover letters, utility requests, lost-and-found notices, ration or pension applications, ride disputes).
• Unseen ID formats. 65% of the ID values (2,386 of 3,650) are rendered in shapes, such as digit groupings or letter-digit splits, that no training example uses; 799 of those use separator styles that never appear in training at all ("strict" below).
• New names. Person names and addresses are written and tagged by the test model itself, not taken from the training name list (email and UPI usernames still use that list).
• Checked twice. Gemma 4 cross-checked every test text for unlabelled personal data (flagged texts dropped). Claude agents then reviewed a stratified sample of 300 texts (LLM review, not human verification); they found a gold label error in 2.1% of spans (mostly missing labels). Re-scoring those 300 texts with the corrected labels left their F1 at 0.960, so label noise does not explain the errors the model makes.
• Overlap with training. 55 of the 1,582 test texts (3.5%) share at least one 8-word sequence with the training texts, all stock phrases such as "kindly process my application at the earliest". None of the 12,000 public benchmark texts used below shares one.

1,582 texts: 1,289 with personal data (6,123 spans) and 293 without.

Results (model + rules)

	Precision	Recall	F1
Exact spans, all labels	0.954	0.964	0.959
Characters masked (any label)	0.985	0.990	0.987

Recall by ID format	Model only	Model + rules
Formats seen in training (1,264 spans)	0.975	0.978
Unseen formats (2,386 spans)	0.793	0.959
Unseen formats, strict subset (799 spans)	0.945	0.983
Values typed by the test model itself, such as names (2,473 spans)	0.963	0.963

Texts with no personal data that got at least one false alarm: 2.4% (7 of 293).

An earlier test set built the same way with a different seed (1,601 texts, used while developing the rules) gives 0.956. One caveat: the IP-address rule was added after the fresh test showed IP addresses as the weakest label (F1 0.667). The rule was built by studying errors on the earlier set, but the IP score of 1.000 below should be read as optimistic.

F1 by label and by language

Label	F1	Label	F1
PAN	0.997	PERSON	0.958
PASSPORT	0.992	DOB	0.953
UPI_ID	0.991	AADHAAR	0.950
EMAIL	0.990	IFSC	0.949
GSTIN	0.984	CARD_NUMBER	0.945
VEHICLE_REG	0.983	PINCODE	0.930
DRIVING_LICENCE	0.976	ADDRESS	0.919
VOTER_ID	0.972	BANK_ACCOUNT	0.879
PHONE	0.965	IP_ADDRESS	1.000

Language	F1	Language	F1
Punjabi	0.982	Kannada	0.962
Urdu	0.980	Malayalam	0.960
Marathi	0.975	Telugu	0.959
Hindi	0.969	Hinglish	0.951
Tamil	0.965	Odia	0.940
Gujarati	0.963	English	0.932
Bengali	0.962

Comparison with other PII models

Every model was run on the same texts. Transformers token classifiers used the transformers pipeline with simple aggregation and a 64-token stride; GLiNER models were prompted with our 18 label names at threshold 0.5; Presidio used spaCy en_core_web_lg with its Indian recognisers switched on and a score threshold of 0.4. Scores are character-level redaction: whatever label a model uses, did it mask the personal-data characters? This makes models with different label sets comparable.

Read this table with two caveats. First, the fresh test set is ours: it shares Chitragupta's label set and synthetic style, so it favours Chitragupta. Second, several of these models tag things outside our label set (cities, organisations, dates) and some are designed to run behind a separate regex layer for IDs; anything they mask outside our labels counts as a false positive here. Recall is the fairer column for those models.

On our fresh test set	Precision	Recall	F1	PII-free texts flagged
Chitragupta (model + rules)	0.985	0.990	0.987	2.4%
hiteshwadhwani/pii-model-indicv2	0.658	0.894	0.758	100%
fastino/gliner2-privacy-filter-PII-multi	0.926	0.627	0.747	61%
openai/privacy-filter	0.673	0.757	0.712	87%
OpenMed/privacy-filter-multilingual-v2	0.719	0.678	0.698	96%
plingampally/meridianpii-hi-v2	0.627	0.716	0.668	98%
knowledgator/gliner-pii-large-v1.0	0.943	0.403	0.565	35%
rotalabs/sakshi-pii-in	0.515	0.489	0.501	98%
Microsoft Presidio (spaCy en_core_web_lg + Indian recognisers)	0.357	0.550	0.433	100%
VrajGoti/privacyguard-ner	0.210	0.926	0.343	100%

On two public benchmarks built by others, Chitragupta is not the best model.

Redaction F1	maskflow-ai/indiapii-bench (2,000 texts)	Ylemis India PII (random 2,000 of 10,000)
knowledgator/gliner-pii-large-v1.0	0.966	0.965
Chitragupta	0.954	0.878
hiteshwadhwani/pii-model-indicv2	0.871	0.963
OpenMed/privacy-filter-multilingual-v2	0.929	0.960
fastino/gliner2-privacy-filter-PII-multi	0.774	0.952
plingampally/meridianpii-hi-v2	0.895	0.923
openai/privacy-filter	0.883	0.917
VrajGoti/privacyguard-ner	0.861	0.835
Microsoft Presidio	0.707	0.819
rotalabs/sakshi-pii-in	0.539	0.686

What these show:

• maskflow: Chitragupta finds almost everything (recall 0.998). Its precision is 0.914 because it tags chat speaker names that the benchmark leaves unannotated, and because it flags 48% of the benchmark's planted look-alikes (timestamps, invoice numbers) as personal data.
• Ylemis: precision 0.999, but recall on addresses is only 0.22. Ylemis addresses are short one-line forms ("803 Park Street, Mysuru 486221"), and Chitragupta was trained on longer, fuller addresses. Every other label is found almost perfectly.

On Ylemis, organisation, location, medical ID and account ID entities were excluded from scoring (account IDs there are internal policy_… references, not bank accounts); on maskflow, ABHA health IDs were excluded. Both benchmarks are synthetic.

Limitations

• Short addresses are often missed. One-line addresses such as "12 Lake Road, Kolkata" are found far less reliably than full postal addresses (the pincode next to them is usually still caught).
• Look-alike numbers. Timestamps, order and invoice numbers can be flagged as bank accounts, cards or driving licences. For redaction this over-masks rather than leaks, but it matters if you use the labels downstream.
• Bank account numbers have no checksum, so they are the weakest ID label (F1 0.879); digit strings near them can be confused with card or Aadhaar numbers.
• Long inputs. The model was trained on texts of a few hundred tokens. On long inputs made by joining six test texts (450–1,400 tokens), exact-span F1 was 0.93–0.94, against 0.96 for the same texts one at a time. For long documents, run it per message or paragraph.
• Synthetic evaluation only. No public human-labelled test set of Indian personal data exists that we could use. The test set was written by an LLM, checked by LLMs and reviewed by LLM agents, not by people. Real text will be messier.
• Names. PERSON covers people only. Names of gods, brands, places and companies are not tagged, and short or ambiguous names in casual chat are a common miss.
• Scope. Health IDs (ABHA), medical details, account usernames, URLs and employee IDs are not covered. The model is not a compliance tool on its own; use it as one layer in a privacy process and review its output where errors are costly.

FAQ

What is Chitragupta?

Chitragupta (arunp0/chitragupta-pii-indic) is an open-source PII detection and redaction model for Indian languages and Hinglish. It finds 18 types of personal data, including Aadhaar, PAN, UPI IDs, phone numbers and names, in 13 languages and returns their character offsets.

What is a good open-source model for detecting PII in Indian languages?

It depends on the text. On our fresh synthetic test set covering 13 languages, Chitragupta had the highest redaction F1 (0.987) of the 10 systems compared. On the public IndiaPII-Bench and Ylemis benchmarks, knowledgator/gliner-pii-large-v1.0 scored higher (0.966 and 0.965 against 0.954 and 0.878). Test on a sample of your own data before choosing.

How do I detect and mask Aadhaar, PAN or phone numbers in Hindi or Hinglish text with Python?

Install torch, transformers and huggingface_hub, then follow the Quickstart: Chitragupta(path).redact(text) returns the text with each value replaced by its label, and detect(text) returns the spans with character offsets.

Which languages and scripts does it support?

Hindi, Hinglish, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and English, in native scripts and in Latin script, including native digits and numbers spoken out as words in transcripts.

Is it free for commercial use?

Yes. The weights and code are Apache-2.0, the base model is MIT, and the training data was generated only with open-weights models.

Does it work offline, on CPU?

Yes. It is a 560M-parameter encoder that runs locally with PyTorch; about 24 ms per short text on a multi-core server CPU in batches of 32. No text leaves your machine.

How is it different from regex rules or Microsoft Presidio?

Regex finds well-formed IDs but misses names, addresses, unusual formats, native digits and spoken numbers. Chitragupta combines a multilingual tagger with checksum rules. On our fresh synthetic test set it masked 99.0% of personal-data characters against 55.0% for Presidio with its Indian recognisers (see Comparison).

Does it make a system compliant with India's DPDP Act?

No model does that on its own. Chitragupta can be one layer for minimising personal data in logs, analytics or LLM prompts; review its output where mistakes are costly and see Limitations.

How accurate is it on real-world text?

Unknown: there is no public human-labelled test set of Indian personal data. On synthetic test data it scores 0.959 exact-span F1 on our fresh set, 0.954 redaction F1 on IndiaPII-Bench and 0.878 on the Ylemis benchmark, where short addresses are its main weakness.

Files

File
model.safetensors, config.json, tokenizer.json, tokenizer_config.json	the tagger (float32)
chitragupta.py	Chitragupta class: detect, detect_batch, redact
chitragupta_rules.py	rule layer
chitragupta_ids.py	checksum and format validators used by the rules (also contains the value generators used to build the data)
LICENSE, NOTICE	Apache-2.0 licence; copyright and the XLM-R MIT notice

Citation

@misc{prabhakaran2026chitragupta,
  title  = {Chitragupta: PII detection for Indian languages and Hinglish},
  author = {Arun Prabhakaran},
  year   = {2026},
  url    = {https://huggingface.co/arunp0/chitragupta-pii-indic}
}

Licence

Apache-2.0. Copyright 2026 Arun Prabhakaran. Built on FacebookAI/xlm-roberta-large (MIT). Training text generated with Gemma 4 and filtered with Qwen3.6, both open-weights models.