kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6

AliceAI-T5-35B-A0.6B · chat + tool-call LoRA, v6

Status: v6, superseded by v6.1. Every spoken reply thinks first (100% of probes and held-out turns) and this version is the steadier one under sampling, but it ignores "do the task, but don't use tools" on half of those probes — v6.1 fixes that and scores higher greedy. Prefer v6.1 unless you decode with sampling.

Only the adapter is published here. The merged upload was blocked by a merge check that later turned out to be a false alarm: it compared teacher-forced loss on the 6 shortest evaluation rows, where bf16 rounding of the merged weights moves individual rows by ±0.03 in either direction. Measured over 40 rows the merge is sound (0.8450 before, 0.8437 after), and the check now uses 40 rows. v6.1 ships a merged model.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model with no chat format. It teaches the model to chat, to call tools when a task needs them, to answer in text when it does not, and to say nothing when there is nothing to say. Every answer it gives thinks first: each non-silent training target starts with a <think>…</think> block.

Adapter	kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6 (this repo)
Merged bf16 weights	kaivoss/AliceAI-T5-35B-A0.6B-chat-v6
Fine-tuning example	finetune_chat_lora.py in this repo
Earlier versions	v5 (best before v6; thinks only sometimes), v4 (fixed repeated calls after tool results; regressed on plain questions), v3, v2, v1 (superseded, known flaw)

How we got here

v1 ran inside an agent harness and misbehaved: a bare "hi" produced 17 to 22 parallel tool calls, "don't read any skills" was ignored, replies were empty or endless. Cause, measured: its chat format used the vocabulary's [INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*] tokens, whose embeddings in the base model are untrained placeholders, identical to the unused <SPAN#5000>. The model could not tell roles apart or emit [REPLY_END], and LoRA on attention cannot teach embeddings. placeholder_ids.json lists all 5,832 placeholder tokens, and a unit test fails if the chat format ever uses one.

v2 switched to plain-text role markers and </s> as the end of a turn, and mixed plain chat into the data. It passed 61% of its behaviour probes (base model: 11%) but still, under a coding-agent prompt, invented a task for a bare "hi" and called a tool, repeated a tool call after its result instead of answering, over-thought simple questions, and answered "thanks" with reasoning instead of silence.

v3 changed only the data: more greetings, small talk and "no tools" instructions under varied agent prompts, real "final answer right after a tool result" turns, reasoning capped at 600 tokens and optional, thank-you silence under harness prompts, and no training target that repeats a recent call. It passed 88% of the (rebuilt) probes, with greetings at 24/24, but under pi's prompt it still repeated the call it had just made instead of answering once a tool result arrived (after-tool probes 4/10). The cause was visible in the data: pi's prompt only ever appeared with plain replies, never next to a tool result.

v4 changed only the data again, to put exactly that into training: real General-Agent trajectory prefixes retargeted onto pi's tools and prompt (about 85% of its calls map onto read/bash/edit/write, but no whole trajectory does, so only the part before the first unmappable call is used), plus templated pi flows with deterministic results and answers. It fixed the bug: after-tool probes went from 8/23 (v3) to 22/23. But it regressed elsewhere: a plain question under pi's prompt got a pointless file read (2/4 greedy), "don't use tools" was ignored in 3 of 8 probes, and on held-out real turns the right-kind-of-reply rate fell from 91% to 83% (it under-called tools). Its sampled score was 10 points below its greedy score, so many decisions were close calls.

v5 changed only the data once more (plain questions under pi's prompt with no instruction, "don't use tools", rebalancing towards tool calls, richer flows). It was the best version so far: 96% right-kind-of-reply on held-out real turns, and greetings, plain questions, "don't use tools" and silence all passed in both greedy and sampled mode. Tried in pi it behaved on the prompts that had broken v1, with one problem: on a story request it looped one dialogue for 13,221 tokens. It also still called a tool after "don't use tools" when the task needed one (2/8), and some answers after tool results were empty (the answer trapped inside <think>) or dropped the key fact.

v6 changes only the data, aimed at those, plus one new rule:

• The rule: every answer thinks first. Every non-silent target starts with <think>…</think>; the data builder rejects any target that does not, and re-checks the finished file. Reasoning is real where a source has it (the agent data: about 100% of action and final turns in Code, General and Search; UltraData's IF-think split) and exact where it is templated (flows, greetings, negations: the logic is known). A real turn with no reasoning is dropped, never given invented reasoning. UltraData's Knowledge traces were unusable (median 5,241 tokens; one in 400 fits a 600-token budget) and smoltalk has none, so neither is used. Silent rows stay empty: there is no answer to think before.
• negtask, about three times more: a task that would need a tool plus "don't use tools", with broader phrasings (count lines, find a file, make a directory, run a script). The right reply is an honest text answer, not a guess and not a call.
• longform: real long answers that end (UltraData IF-think), so long text learns to stop. Probes check for loops and endings.
• 133 probes (up from 129) with long-form probes, a loop check, and a think-rate metric on probes and held-out turns.

Chat template

The base repo ships no chat template; chat_template.jinja defines one inside the model's UL2 frame:

encoder:  [_S_]<|system|>…\n<|user|>…\n<|assistant|>…\n<|tool|>…\n<|assistant|><SPAN#0>
decoder:  <s><SPAN#0>  →  <think>reasoning</think>text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call></s>

Tools go in the system message as text in the GLM format (# Tools … <tools>{json per line}</tools> …), produced by glm_tools.py. Reasoning always comes first (capped at 600 tokens in training). An empty reply is just </s>. Pass alice_frame=False to apply_chat_template when a runtime adds [_S_] and <SPAN#0> itself (vLLM plugin below).

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_output

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                     do_sample=True, temperature=0.6, top_p=0.95, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
reasoning, content, tool_calls = parse_output(clean_decode(tok, gen))

Use sampling, not greedy: greedy decoding looped in testing.

Serving with vLLM and an OpenAI-compatible client

The community plugin (AntonProkopyev/AliceAI-T5-35B-A0.6B-vLLM) loads the merged repo unchanged, but it only serves /v1/completions and adds the UL2 frame itself. alice_openai_shim.py sits in front of it and provides /v1/chat/completions with tools, reasoning and streaming (simulated). It also samples by default (the harness sends no temperature), merges duplicate tool calls, caps parallel calls, never lets a turn continue as the user, retries an empty reply after a user or tool message, shows an answer that landed inside <think> as the reply, and cuts a repetition loop after its first cycle (v5 once looped a story for 13,221 tokens). serve_smoke.py exercises it with the openai Python client.

Training

Method	LoRA (bf16 base; not QLoRA, not full fine-tuning)
Rank / alpha / dropout	16 / 16 / 0.05 (scale 1.0)
Targets	q_proj, k_proj, v_proj, o_proj in encoder self-attn, decoder self-attn and decoder cross-attn (experts are fused parameters, not nn.Linear)
Steps	200 (batch 1 × 8 accumulation, each example seen once)
Optimiser	AdamW, lr 2e-4, cosine, 10 warmup steps, bf16, gradient checkpointing
Context	up to 32,768 tokens; longer examples are skipped per turn, never truncated, and turn choice is fit-aware so no row is lost
Attention	SDPA (the remote code does not declare it; enable_sdpa() sets the flag, checked against eager on padded batches)
Hardware	1 × H200

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn, then </s>.

category	examples	mean tokens	max tokens	reply kinds
agent_action	520	13,029	30,504	{'action': 520}
agent_final	140	11,301	26,732	{'final': 140}
agent_plain	100	6,534	16,390	{'plain': 100}
chat_plain	110	338	1,939	{'plain': 110}
chat_tools	80	4,233	7,585	{'plain': 80}
eval_agent	100	13,889	28,946	{'action': 82, 'plain': 18}
eval_chat	40	567	1,493	{'plain': 40}
harness	70	2,162	6,058	{'plain': 70}
negation	130	1,529	1,907	{'plain': 130}
pi_flows	330	1,510	2,075	{'final': 163, 'action': 167}
pi_plain	130	1,977	3,432	{'plain': 130}
pi_retarget	250	7,274	27,093	{'action': 250}
silent_agent	50	10,807	24,808	{'silent': 50}
silent_synth	90	1,377	2,024	{'silent': 90}

Sources: real agent trajectories (UltraData-SFT-Agent-2609, GLM-formatted copy linked above), plain Q&A from UltraData-SFT-2605 (English IF and Knowledge), greetings and system-prompted chats from smoltalk, a sanitized capture of the pi coding-agent prompt and tool schemas (harness_pi.json), General-Agent trajectories retargeted onto pi's tools, templated pi flows with exact reasoning, and negation-with-task replies (pi_flows.py), and templated greetings, small talk, "no tools" instructions and thank-yous (phrases.py). Targets are filtered to at most 8 parallel calls, 600 reasoning tokens and no repeat of a recent call. Held-out rows are never trained on.

Evaluation

Behaviour probes (probes.py): greetings, "don't use tools", small talk and plain questions under pi-style prompts (with and without a 14-skill list, and with a tool subset) and a non-coding weather / calculator / search assistant; real first calls; answering after a tool result; the second call of a multi-step task; silence after a thank-you or a finished task. A probe passes only if every expectation holds, the reply finished, and it never wrote another turn. The previous adapter (v3) is scored on the same probes.

probe category	this adapter, greedy	this adapter, sampled	previous adapter (v5), greedy
action	18/22	19/22	19/22
after_tool	30/31	29/31	25/31
greeting	23/24	23/24	24/24
longform	2/4	4/4	2/4
mid_step	2/3	1/3	2/3
neg_question	4/4	4/4	4/4
neg_task	2/8	5/8	1/8
negation	8/8	8/8	8/8
plain_question	8/8	8/8	8/8
silent	15/15	15/15	15/15
smalltalk	6/6	6/6	6/6
thinks before answering (non-silent probes)	100%	100%	5%
all probes passed	89%	92%	86%

Held-out real turns (this adapter):

n = 60 held-out rows, greedy, 1,024 new tokens.

metric	adapter
generations cut off at the token cap (not judged below)	13.3%
right kind of reply (tool call / text / nothing)	92.3%
... when the right reply was a tool call	94.4%
... when the right reply was text	87.5%
... when the right reply was nothing	n/a
first tool name matches the real one	61.8%
thinks before answering	100.0%
wrote another turn (`<	user
turns with duplicate tool calls	1.7%
most tool calls in one turn	11

These are small samples (133 probes, a few dozen held-out rows) of greedy or single-sample generations. They check for the specific failures above; they are not a capability benchmark. Tool-call arguments are not checked, only the tool name, and the tool-name metric counts a reasonable alternative as wrong.

Limitations

• Small budget. 200 steps is 1,600 examples: enough to move behaviour, not to add agentic competence the base lacks.
• Attention-only adaptation. Experts and embeddings are untouched.
• Templated data. Greetings, small talk, negations, thank-yous and the pi flows are templated, which is why a hash-held-out 20% of their wording is kept for the probes; real-world phrasing outside those templates is untested.
• Tokenizer quirk. The tokenizer inserts a space after every newline when decoding; use clean_decode from alice_chat.py (important for code in tool arguments).
• Data. Reasoning traces come from whatever model produced the upstream data, not from AliceAI. Plain chat comes from English-only sources. No safety tuning was done: the model can and will run bash commands if a harness lets it.
• vLLM. The merged-weights path was verified with v1's checkpoint on the community plugin; re-verify for this one.

Licence and attribution

Apache-2.0, matching the base model (© Yandex LLC) and the data (UltraData: arXiv:2602.09003; smoltalk: Apache-2.0 for its new subsets).