train-jev-27b-lora v0.4.0

Qwen3.8-27B + LoRA + a 24-slot FP32 decision head, returning probabilities for yes/no (noul), multiple-choice (choice) and rating (score) questions without generating text. Uses standard causal attention and the train-jev System One API.

The larger model reaches 79.52% train-jev-pk accuracy and 57.01 full public Decision Index (previous release: 78.72% / 55.58). This is a trade-off, not a universal upgrade: transfer-v4 accuracy drops 85.8232% β†’ 84.7561%, and ECE worsens 0.031566 β†’ 0.068090.

For a smaller model, see shibing624/train-jev-4b-lora.

Use

git clone https://github.com/shibing624/train-jev.git && cd train-jev
pip install -r requirements.txt
python -m trainjev.serve --base Qwen/Qwen3.8-27B \
  --adapter shibing624/train-jev-27b-lora --port 8020

The first run downloads the backbone, adapter, head.pt and saved temperatures. Plain PEFT loading does not apply the decision head or calibration; use train-jev, not a generative-chat pipeline. Pin this release with --adapter_revision v0.4.0. See the serving guide for the API.

Evaluation

Ours are the current weights. Other models are historical references, not rerun or newly matched in this round.

Evaluation train-jev-4b-lora train-jev-27b-lora Kev-4B (historical) autotrust/JEV-27B-VL (historical) TypeSafe Jev v1.13 (historical API)
train-jev-pk accuracy, 2,494 questions ↑ 76.03% 79.52% 63.50% 73.47% –
train-jev-pk ECE15 ↓ 0.0930 0.0901 0.159 0.107 –
Classification PK accuracy, 1,300 questions ↑ 0.6523 0.6938 0.640 0.690 0.700
Classification PK ECE10 ↓ 0.0622 0.0377 0.083 0.094 0.130
Kev transfer-v4 accuracy, 656 scored questions ↑ 0.8201 0.8476 0.817 0.852 0.857
Kev transfer-v4 ECE10 ↓ 0.0588 0.0681 0.042 0.097 –
Decision Index historical sample, 6,000 requests ↑ 43.84 57.54 38.4 54.1 59.7
Decision Index full public panel, 110,087 requests ↑ 42.00 57.01 – – –

Exact metrics, prior-release comparison, benchmark regressions and evidence hashes are in metrics.json; protocols and context are in the results guide.

  • PK uses 22-task macro accuracy and per-task upper-inclusive ECE15; classification uses pooled upper-inclusive ECE10. Transfer evaluated 764 original requests and scores 656 clean knowable questions, preserving Kev normalization and its ECE10 boundaries.
  • The historical sample has 6,000 requests / 47,463 child answers, 41 input catalogs and 35 scoring benchmarks. Full public evaluation has 110,087 requests / 272,229 children over 37 scoring benchmarks. Six unindexed diagnostics were not requested: canonical suite43 complete=false is retained. These are composite DI scores, not accuracy or the private Full leaderboard.
  • Prior-release controls are original stored predictions and CPU recompositions, not new same-stack inference. Old classification wide choices used tournament; current weights use native single-pass readout.
  • 4B classification and transfer ECE also worsen. Negative DI benchmark deltas are retained in metrics.json, not hidden by aggregate gains. Historical official/API values without matching raw proof are contextual references, not new verified runs.

Training

Rank16 / alpha32 LoRA and the 24-slot head are trained; the backbone is frozen. Initial Jev-distribution distillation is followed by gold/rule refinement and 12,947 admitted tasksource rows across nine sources, including 2,692 native soft targets. Sources include CommonsenseQA, CosmosQA, DynaSent rounds 1/2, WNUT17, LexGLUE CaseHOLD, Fig-QA, QuaRTz and Amazon Counterfactual English.

This release selects step 512 on a 2,048-step scheduler, seed17, batch128. The tasksource stage replaces 32 same-kind choice slots per batch; earlier gold mixing occurs before replacement and is not the final all-draw fraction. Actual 0–512 exposures are recorded in training_recipe.json, not estimated from pool proportions. Calibration is retained byte-identically and was never refitted to these results.

The training guide explains the reusable public-data workflow. Processed training data, generators, optimizer/RNG and raw evaluation predictions are not bundled; exact one-command training reproduction is not provided.

Limitations and attribution

  • PK includes training-source and shared-rule-family items with previous selection exposure; it is not a wholly unseen-source independent final test.
  • After generation/verification, a schema probe displayed native labels in two PK provenance records. They were not used for generation, oracles, screening, selection or tuning, but strict zero-label-access was not fully preserved. No 100% contamination-free claim is made.
  • No new 27B final, SDK, MemWrite8 or vision test is claimed. Earlier 4B evidence is archival; API support does not establish new-weight quality or GPU/HTTP bitwise equivalence.
  • Only choice rows A–P are trained; the extra 240 rows for wide choices are untrained base readouts. Confidence is not a universal correctness guarantee.

Adapter/head and code: Apache-2.0. Upstream data retains its own terms and attribution; see LICENSE and THIRD_PARTY_ATTRIBUTIONS.txt. Independent work, not affiliated with TypeSafe AI, autotrust or upstream authors.

Downloads last month
62
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for shibing624/train-jev-27b-lora

Base model

Qwen/Qwen3.8-27B
Adapter
(371)
this model