Thanushttk/pocket-tts-multilingual

Multilingual Pocket TTS

This is a community finetune of Kyutai Pocket TTS, not an official Kyutai release. It is a first try at one multilingual SentencePiece tokenizer for Pocket TTS.

A 6 layer student distilled from a 24-layer teacher. The tokenizer is SentencePiece, 4,000 pieces. It speaks English, Hindi, Marathi, Kannada, Tamil, Malayalam, and Telugu, and it clones a voice from a short wav. Audio is 24 kHz. The weights are about 418 MB, and CPU is the intended way to run them. Guidance is already baked in, so generate with 1 step and temperature 0.3.

Bengali, Gujarati, Odia, and Punjabi are not supported yet.

The paper is Pocket TTS. The base model is kyutai/pocket-tts, and the code is kyutai-labs/pocket-tts. The safetensors bundle includes Kyutai's codec weights, used here under CC-BY-4.0.

Mix the languages

Write the sentence. Code mixing works, including English mixed with Hindi, and with Tamil, Marathi, Kannada, Telugu, and Malayalam.

Hello, नमस्ते, how are you?

வணக்கம், nice to meet you
आज का plan is simple

Samples

These are generations from this model, not the benchmark set.

English

Life doesn't always go the way we plan, but sometimes the unexpected moments turn out to be the most beautiful ones.

Hindi

ज़िंदगी हमेशा हमारी सोच के हिसाब से नहीं चलती, लेकिन कभी-कभी अचानक मिले पल ही सबसे खूबसूरत यादें बन जाते हैं।

Marathi

आयुष्य नेहमी आपल्या मनासारखं घडत नाही, पण कधी कधी अनपेक्षित क्षणच आयुष्यातल्या सगळ्यात सुंदर आठवणी बनतात.

Kannada

ಜೀವನ ಯಾವಾಗಲೂ ನಾವು ಅಂದುಕೊಂಡಂತೆ ನಡೆಯುವುದಿಲ್ಲ, ಆದರೆ ಕೆಲವೊಮ್ಮೆ ಅನಿರೀಕ್ಷಿತ ಕ್ಷಣಗಳೇ ನಮ್ಮ ಜೀವನದ ಅತ್ಯಂತ ಸುಂದರ ನೆನಪುಗಳಾಗುತ್ತವೆ.

Tamil

வாழ்க்கை எப்போதும் நாம் நினைத்தபடி போகாது, ஆனால் சில நேரங்களில் எதிர்பாராத தருணங்கள்தான் நம் வாழ்க்கையின் அழகான நினைவுகளாக மாறுகின்றன.

Malayalam

ചിലപ്പോൾ ജീവിതത്തിൽ നമ്മൾ ഒരുപാട് ആഗ്രഹിച്ച കാര്യങ്ങൾ നടക്കില്ല, പക്ഷേ പിന്നീടൊരിക്കൽ തിരിഞ്ഞുനോക്കുമ്പോൾ അതെല്ലാം നല്ലതിനായിരുന്നെന്ന് തോന്നും.

Telugu

జీవితం ఎప్పుడూ మనం అనుకున్నట్టు సాగదు, కానీ కొన్నిసార్లు అనుకోకుండా ఎదురయ్యే క్షణాలే మన జీవితంలో అందమైన జ్ఞాపకాలుగా మిగిలిపోతాయి.

Hindi + English

ज़िंदगी में सब कुछ हमारे plan के according नहीं होता, लेकिन कभी-कभी things work out better than we expected.

Use

Install the library, or call it with uvx.

pip install pocket-tts

weights_path and tokenizer_path in the config are relative. Download this repo and run from that folder:

import scipy.io.wavfile
from pocket_tts import TTSModel

model = TTSModel.load_model(config="config.yaml")
voice = model.get_state_for_audio_prompt("my_voice.wav")
audio = model.generate_audio(voice, "Hello, नमस्ते, how are you?")
scipy.io.wavfile.write("out.wav", model.sample_rate, audio.detach().cpu().numpy().squeeze())

The same config on the Hub:

model = TTSModel.load_model(config="hf://Thanushttk/pocket-tts-multilingual/config.yaml")

That address loads the yaml. The weight and tokenizer paths inside it stay relative, so model.safetensors and tokenizer.model still have to sit in the working directory. config.yaml and student.yaml are the same file.

uvx pocket-tts generate \
  --config config.yaml \
  --voice my_voice.wav \
  --text "Hello, नमस्ते, how are you?" \
  --output-path out.wav

Results

64 clips per language from the Sarvam dub benchmark, used here as the eval set. The model clones reference_audio and speaks target_text. Each row is one language. Code-mixing works, and these numbers do not include it.

Word error is Whisper large-v3 for English and Indic Transcribe for the other languages. Prompt similarity is WavLM (microsoft/wavlm-base-plus-sv) between the generated clip and the prompt. UTMOS is a 1–5 quality estimate.

Language	Clips	ASR	Word error	Prompt similarity	UTMOS
English	64	Whisper large-v3	6.2%	0.796	4.25
Hindi	64	Indic Transcribe	10.5%	0.904	3.62
Marathi	64	Indic Transcribe	5.8%	0.911	3.64
Tamil	64	Indic Transcribe	11.0%	0.912	3.51
Telugu	64	Indic Transcribe	13.8%	0.897	3.70
Kannada	64	Indic Transcribe	14.7%	0.894	3.77
Malayalam	64	Indic Transcribe	23.7%	0.863	3.63

English has a lower word error than Hindi and Tamil. Among the Indic languages, Marathi, Hindi, and Tamil are the clearest. Malayalam is the weak language(I didn’t have enough data). English voice match on these dub prompts is lower than the Indic languages.

Model

Architecture	Pocket TTS flow LM, 6 layers, distilled from 24
Text	One SentencePiece tokenizer, 4,000 pieces, 7 languages
Sample rate	24 kHz
Decode	1 step, temperature 0.3
Size	About 418 MB
Device	CPU

Happy to collaborate. The next step is every Indian language, still small enough to run locally where a fast network or a high-end GPU is not around.

If you like this model, give it a like, and follow me for future updates: https://www.linkedin.com/in/thanush-thankachan-255b31148/