akakaiy/MOSS-Audio-8B-Thinking-ru

MOSS-Audio-8B-Thinking — Russian LoRA (ru2, r1) + GGUF

RU коротко: 28.10 добавлена вторая версия — ru2 (рекомендуемая). Русская LoRA для MOSS-Audio-8B-Thinking. Модель лучше базы слышит русскую речь (шум, телефон, дальний микрофон) и русские песни. В отличие от r1 (лежит здесь же), на нерусских песнях она пишет текст на их языке и не сочиняет русский. Внутри: GGUF Q6 для CrispASR (ru2, r1, база), LoRA (PEFT) и патч CrispASR. Включайте CRISPASR_MOSS_CONTENT_ONLY=1 (см. ниже): без него CrispASR подаёт модели 30 с добивки, и на коротких клипах дообученная модель может промолчать.

MOSS-Audio listens to the whole sound — music, tone of voice, background, several speakers. ru2 improves its Russian speech and Russian song transcription over the base model and keeps non-Russian songs in their own language.

Files

file	size	what
moss-audio-8b-thinking-ru2-q6_k-encq8.gguf	7.86 GB	base + LoRA merged (NF4-aware, see Training), Q6_K LLM, audio-encoder linears in Q8_0
lora-ru2/	381 MB	PEFT LoRA (r=32, alpha=32) for OpenMOSS-Team/MOSS-Audio-8B-Thinking
crispasr-moss-decode.patch	9 KB	CrispASR moss-audio patch: content-only audio path, token budget, think budget, loop breaker (updated 28.10; a superset of the r1 patch)
moss-audio-8b-thinking-ru-r1-q6_k-encq8.gguf, lora-r1/		the previous version (22.10)
moss-audio-8b-thinking-q6_k-encq8.gguf	7.86 GB	the untouched base in the same format (for A/B)

The GGUF runs in CrispASR (--backend moss-audio); ~7.7 GB VRAM, fits a 10 GB RTX 3080.

Results

Measured on one RTX 3080, GGUF Q6 through patched CrispASR with CRISPASR_MOSS_CONTENT_ONLY=1 (loop breaker on). None of the test sets was trained on. CER, lower is better.

set	Whisper large-v3	GigaAM v3 (e2e RNN-T)	MOSS base	MOSS r1	MOSS ru2 (this)
FLEURS-ru, 47 clips without digits in the reference	1.02 %	1.01 %	3.80 %	3.31 %	3.55 %
dev-300 (held-out: noisy / phone / far-field / crowd / podcast)	10.08 %	4.93 %	11.38 %	7.11 %	7.46 %
Russian songs, 30 full-mix chunks of held-out artists	69.2 %	17.2 %	35.0 %	25.8 %	24.6 %
jam-alt (311 en/es/de/fr song chunks)	—	—	19.1 %	—	22.0 %
jam-alt: answers mostly in Cyrillic (invented Russian)	—	—	0 / 311	—	1 / 311

Whisper and GigaAM run outside MOSS, so the audio-path fix does not touch their numbers. The r1 numbers published on 22.10 were measured before the fix (30-s padding); here r1 and the base are re-measured on the fixed path.

Listening skills (Sound and voice-tone prompts with reasoning):

	base	ru2
ESC-50 fold 5 (100 clips): sound class named	67 %	64 %
ESC-50: speech / lyrics invented on non-vocal clips	1 / 86	1 / 86
RESD test (140 clips): emotion named first == label	27 %	24 %
45 description prompts: unfinished or skipped reasoning	0	0

Honest summary.

• On noisy and spontaneous Russian speech, and on Russian singing over music, ru2 is clearly better than the base and than Whisper.
• On clean read speech, dedicated ASR (GigaAM, Whisper) is still ~3× better.
• Sound and voice-tone description is about at the base level, slightly lower.
• The intended use is a hybrid: a fast ASR writes the words, and MOSS listens to how it sounds. ru2 can take the ASR text in the prompt.

ru2 vs r1:

• ru2 does not write Russian on foreign songs; r1 does.
• ru2 is slightly better on Russian songs and slightly worse on clean speech.

Limitations

• On clean read speech it is behind GigaAM and Whisper (see above).
• It sometimes normalises spontaneous speech (drops fillers like «ну», «э»).
• Voice-tone / emotion recognition is not improved over the base.
• Only Russian (plus non-Russian songs) was measured.

Training

• QLoRA on one RTX 3090: LLM in NF4 (bitsandbytes, double quant, block 64), audio encoder / adapter / lm_head in bf16. LoRA on all LLM attention + MLP projections and on the audio encoder's q/k/v.
• Merge note. The adapter was trained against the NF4 base. The GGUF was merged into NF4-round-tripped LLM weights. A naive merge into the bf16 base loses about 1 CER point on FLEURS. With PEFT, load the base in 4-bit NF4 for the same results.
• Stage 1 (from the base, 1 epoch, lr 2e-4): ~490 h of public Russian speech (Common Voice 21, PotatoHD ru-asr-audio, SOVA, Russian LibriSpeech, Sber Golos subsets, Taiga, Podlodka train). References are re-cased where GigaAM agreed. Plus Russian song chunks and ~2 % self-distilled base-model descriptions.
• Stage 2 (on top, 1 epoch, lr 5e-5), 103 k rows:
  • verbatim cased transcripts of spontaneous speech (punctuation restored by an LLM under an exact-words lock);
  • the base model's own transcriptions of 830 non-Russian song chunks (FMA vocal genres), under both a lyrics prompt and a speech prompt;
  • Russian songs, cased read speech, lowercase speech, base-model descriptions.
• The training audio itself is not published.

Usage (CrispASR server)

# build CrispASR with the patch (git apply crispasr-moss-decode.patch), then:
CRISPASR_MOSS_CONTENT_ONLY=1 CRISPASR_MOSS_MAX_NEW=1024 CRISPASR_MOSS_THINK_BUDGET=640 CRISPASR_MOSS_LOOP_BREAK=6 \
crispasr --server --port 11443 --backend moss-audio -m moss-audio-8b-thinking-ru2-q6_k-encq8.gguf -n 1024
# transcription: CRISPASR_MOSS_MAX_NEW=768 CRISPASR_MOSS_THINK_BUDGET=256
curl -F file=@voice.ogg -F response_format=json \
     -F prompt="Transcribe the speech in this audio verbatim, in the original language. Output only the transcript." \
     http://127.0.0.1:11443/v1/audio/transcriptions
# song lyrics: "Transcribe the lyrics of this song verbatim, in the original language. Output only the lyrics."

Patch env knobs:

• CRISPASR_MOSS_CONTENT_ONLY=1: feed the LLM only the real audio frames, as the reference Python processor does. Upstream pads every clip to 30 s; on that path fine-tuned adapters may answer nothing on short clips.
• CRISPASR_MOSS_MAX_NEW: upstream hard-codes 512 and ignores -n.
• CRISPASR_MOSS_THINK_BUDGET: closes <think> after N tokens.
• CRISPASR_MOSS_LOOP_BREAK: stops on N repeats of the same n-gram.
• CRISPASR_MOSS_REP_PENALTY.

With transformers + PEFT: load the base with its own code (trust_remote_code, LLM in 4-bit NF4), then PeftModel.from_pretrained(model, "<this repo>", subfolder="lora-ru2").

License: follows the base model's license.