DuplexJev-Gemma-31B

Archived. Superseded by DuplexJev-32B-Para. This model has no turn-taking training; 32B-Para is the recommended server model. This repository stays online for existing users and paper reproducibility, but is no longer updated.

📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev

Built on Gemma-4-31B-it + the Qwen3-ASR-0.6B audio encoder. The non-Qwen member of the DuplexJev family: strongest English reasoning of the LLMs we tested, Apache-2.0 base.

DuplexJev reads typed, closed-set decisions from speech (has the user finished? who is speaking, in what mood?) as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (encoder + trained connector + LLM, bf16, about 58 GB) and runs on vLLM with a small plugin.

Audio encoder (frozen)	Qwen3-ASR-0.6B encoder, 12.5 frames/s
Connector (trained)	2 frames stacked → 6.25 audio tokens/s → SwiGLU MLP → LLM width; audio embeddings ×3 and wrapped in Gemma's native <|audio> … <audio|> markers
LLM	Gemma-4-31B-it text model. Unlike DuplexJev-32B / 4B, the LLM is not frozen: a LoRA (r 16, α 32, q/k/v/o) was trained in the paralinguistic stage and merged into the weights
Training	content alignment (transcript distillation, LLM frozen) → LoRA content warm-up → gender + emotion with LoRA under the mixed objective (CE on decisions, KL on content)
Inputs	audio + text. Gemma's vision tower is not included.

Serve with vLLM

pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.2.0"
vllm serve adventists-ai/DuplexJev-Gemma-31B --max-model-len 4096

One 80 GB GPU is enough (or --tensor-parallel-size 2). duplexjev-vllm ≥ 0.2.0 registers DuplexJevGemmaForConditionalGeneration (Qwen3-ASR encoder + connector + ×3 scale + audio markers; the LLM runs on vLLM's own Gemma kernels). The placeholder in prompts stays <|audio|>; the plugin expands it.

Each question is one chat request with max_tokens=1: list the options under letters and read the log-probabilities of the letters. A ready-made client is examples/vllm_client.py:

from vllm_client import DuplexJevClient

dj = DuplexJevClient("http://localhost:8000/v1")
dj.decide(open("call.wav", "rb").read(), {
    "gender":  ("What is the perceived gender of the speaker?", ["female", "male"]),
    "emotion": ("What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"]),
})

The prompt format the connector was trained on:

The user said: <|audio|>

Question: What is the perceived gender of the speaker?

Options:
A. female
B. male

Answer with only the letter of the correct option.

For Chinese speech ask in Chinese: 用户说：<|audio|>, 问题：…, 选项：, 请只回答正确选项的字母。

Results

Served by vLLM 0.29 with duplexjev-vllm 0.2.0, questions in the training wording above (one H200, bf16; % correct):

model	qa100	ZJU-ML	Easy-Turn	gender	emotion
DuplexJev-Gemma-31B (this repo)	97	87	68.0*	94.4	90.6
DuplexJev-32B	94	88	79.4	89.8	89.9
DuplexJev-4B	77	49	60.0	87.9	90.9

* No turn-taking data was used for this model; Easy-Turn is zero-shot here (DuplexJev-32B saw the Easy-Turn training split).

Wording sensitivity. With a question wording that differs from training (our paper-protocol evaluation script), this model scores qa100 96, ZJU-ML 84, but gender 78.5 and emotion 60.9. The DuplexJev-32B connector (frozen LLM) is much less sensitive to wording. Use the prompt format above.

qa100: 100 bilingual spoken multiple-choice questions (adventists-ai/qa100). ZJU-ML: main-language part of the ZJU audio benchmark v2.0.0. Easy-Turn: 800-item four-way turn-state test set. Gender: 800 real utterances (AISHELL-1, LibriSpeech). Emotion: 800 utterances (ESD, CREMA-D).

Limitations

• Evaluated on short read or acted speech; not on streaming input.
• Paralinguistic accuracy depends on prompt wording (see above).
• No turn-taking training yet; prefer DuplexJev-32B for end-of-turn decisions.
• Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.

License

CC BY-NC 4.0, research use only. The connector and LoRA were trained on the ESD emotional speech corpus, which is licensed for research purposes only, and on CREMA-D (ODbL). The base models keep their own licences: Gemma-4-31B-it (Apache-2.0, Google) and Qwen3-ASR-0.6B (Apache-2.0, Alibaba Cloud). The LLM weights in this repository are Gemma-4-31B-it's text-model weights with a LoRA merged in; the vision tower is not included.

Citation

@misc{jin2026duplexjev,
  title         = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author        = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  year          = {2026},
  eprint        = {2610.02638},
  archivePrefix = {arXiv},
  note          = {Submitted to IEEE ICASSP 2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.