qwen2.5-1.5b-weather-intent-v2

QLoRA adapter for Qwen/Qwen2.5-1.5B-Instruct β€” a production-hardened weather-intent parser (v2). Maps a user message to a structured JSON intent, and rejects what it can't answer instead of confabulating.

v2 over the v1 SFT baseline (which hit 98.6% exact-match but confabulated on out-of-scope input) adds three independently-measured levers:

  • reject training β€” an explicit intent: "out_of_scope" class + boundary-negative data (fixes confabulation)
  • grammar-constrained decoding β€” XGrammar JSON-schema, making out-of-vocabulary values impossible (fixes surface form)
  • a deterministic gate β€” route() decides accept vs escalate from the structured output (model-cascade)
{"intent": "weather", "location": "Paris", "date": "this_weekend", "metric": "rain", "time_of_day": null}
{"intent": "out_of_scope", "location": null, "date": null, "metric": null, "time_of_day": null}

Schema

field values
intent weather, out_of_scope
location place name, or null
date today, tomorrow, day_after_tomorrow, this_weekend, next_weekend, next_week, a weekday, or null
metric temperature, rain, snow, wind, humidity, uv, cloud, general
time_of_day morning, afternoon, evening, night, or null

Results (300 held-out; base few-shot, ft/ft+grammar zero-shot; greedy)

metric base ft ft_grammar
in-scope slot exact-match 52.0% 99.6% 100.0%
OOS recall (rejects unanswerable) 80.0% 100.0% 100.0%
FAR β€” false-accept / confabulation 20.0% 0.0% 0.0%
FRR β€” false-reject 20.9% 0.4% 0.0%
out-of-vocab errors 30 1 0

The honest read: reject-training does the heavy lifting (ft alone drops false-accept to 0); grammar is the last-mile guarantee that drives out-of-vocab to a structural 0. The gate turns structure into a cheap-over-escalation routing decision. Grammar guarantees validity, not correctness β€” the constraint tax is measured, not assumed.

Training

  • QLoRA (4-bit NF4, double-quant, bf16), LoRA r=16 / alpha=32 on all linear layers
  • 1 epoch, effective batch 16, LR 0.0002 cosine, assistant-only loss
  • 1500 examples (25% out-of-scope, 4 balanced strata); single RTX 3050 Ti (4 GB)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, device_map="auto"), "Nicholas55555/qwen2.5-1.5b-weather-intent-v2")

sys = "You extract structured intent from user messages about weather. Return ONLY a JSON object..."
msgs = [{"role": "system", "content": sys}, {"role": "user", "content": "how's the pollen in Austin?"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(tok.decode(model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=64)[0], skip_special_tokens=True))
# -> {"intent": "out_of_scope", ...}   (pollen is not a supported metric)

For guaranteed-valid output, pair with XGrammar constrained decoding against the schema.

Downloads last month
18
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Nicholas55555/qwen2.5-1.5b-weather-intent-v2

Adapter
(1,772)
this model