qwen35-9b-takehome-merged

Merged full checkpoint on Qwen/Qwen3.5-9B (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a), part of a multi-expert training and model-merging experiment. Target: all four (SWE-bench Pro v2-hard, Terminal-Bench 2.0, HLE, ASI-Bench). Code, configs, logs and the full protocol: https://github.com/ktwu01/qwen36-35b-takehome

Training

QLoRA (nf4 base, bf16 compute), r=32, alpha=64, dropout 0, targets: full-attention q/k/v/o_proj, Gated DeltaNet in_proj_qkv/in_proj_z/out_proj, MLP gate/up/down_proj (language model only), lr 1e-4 cosine, 3% warmup, global batch 16 (union: 15), 16,384-token context, 1 epoch, assistant-only loss, seed 20261018.

Data

  • Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k (MIT)
  • open-thoughts/OpenThoughts-Agent-v1-SFT (Apache-2.0)
  • nvidia/OpenScienceReasoning-2 (CC-BY-4.0, NVIDIA)

Decontaminated with 13-gram overlap against all four benchmarks; see data/README.md in the code repo.

Merge recipe

{
 "name": "ties-d0.5-l1.0",
 "merge_args": "--method ties --experts outputs/swe outputs/term outputs/sci --density 0.5 --lam 1.0",
 "dev_mean_token_loss": 0.32781994345410975,
 "dev_token_loss": {
  "swe": 0.22026807381601063,
  "term": 0.1873204659077618,
  "sci": 0.5758712906385569
 },
 "rule": "lowest mean dev token loss over swe/term/sci among 3-expert merges; tie -> lower worst domain",
 "ranking": [
  {
   "name": "ties-d0.5-l1.0",
   "mean": 0.32782,
   "worst": 0.57587
  },
  {
   "name": "ties-d0.2-l1.0",
   "mean": 0.32953,
   "worst": 0.57779
  },
  {
   "name": "ties-d0.5-l0.7",
   "mean": 0.3317,
   "worst": 0.57579
  },
  {
   "name": "ta-l0.5",
   "mean": 0.33197,
   "worst": 0.58267
  },
  {
   "name": "dare-d0.5-l1.0",
   "mean": 0.3363,
   "worst": 0.58989
  },
  {
   "name": "ta-l0.7",
   "mean": 0.33852,
   "worst": 0.59366
  },
  {
   "name": "ta-l1.0",
   "mean": 0.36116,
   "worst": 0.62241
  }
 ]
}

Loading

vllm serve ktwu01/qwen35-9b-takehome-merged --quantization fp8 --kv-cache-dtype fp8 --max-model-len 262144 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

or AutoModelForCausalLM.from_pretrained("ktwu01/qwen35-9b-takehome-merged") (same architecture and files as the base).

Evaluation

Status: final data (runners stopped 2026-10-19 20:20 CDT); Terminal-Bench seeds 2-3 incomplete, one complete seed per model. Experiment-specific protocol (apptainer containers, FP8 weights and KV cache, thinking on, temperature 1.0); not official leaderboard numbers. Full tables, per-task results and failure analysis: https://github.com/ktwu01/qwen36-35b-takehome (README section 6, analysis/failures.md).

Benchmark This model Base (Qwen3.5-9B) Merged - Base, pp (95% CI)
SWE-bench Pro v2-hard (HARD-51, resolved %) 0.00 5.88 -5.88 [-13.73, +0.00]
Terminal-Bench 2.0 (89 tasks, success %) 12.36 (1 complete seed) 19.10 (1 complete seed) -6.74 [-14.61, +1.12]
HLE (2,500 questions, accuracy %) 8.16 7.08 +1.08 [-0.16, +2.32]
ASI-Bench (236 instances, official mean %) 0.51 5.78 -5.13 [-6.70, -3.69]

Macro average over the four benchmarks, merged - base: -4.17 pp (95% CI [-6.92, -1.61]); the experiment's +3 pp target is not met.

ASI-Bench ran in mini-swe-agent's tool-calling mode; this model answers with text bash blocks, which that mode rejects (RepeatedFormatError in 206 of 240 runs), so its ASI score mostly reflects the protocol mismatch. Terminal-Bench has fewer than the planned 5 seeds.

License and limitations

Apache-2.0 (as the base model). Training data licenses above; OpenScienceReasoning-2 is CC-BY-4.0 and requires attribution to NVIDIA. Research artifact: evaluated under the experiment-specific protocol in the code repository (apptainer environment, FP8 serving); scores are not official leaderboard results.

Downloads last month
10
Safetensors
Model size
10B params
Tensor type
BF16
ยท
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ktwu01/qwen35-9b-takehome-merged

Finetuned
Qwen/Qwen3.5-9B
Adapter
(1,123)
this model