Qwen3-0.6B-Weather-Geo-Specialist

Model Description

This model is a LoRA fine-tuned specialist for weather and geolocation tool calling, built with the techniques covered in Chapter 13 of the book "Rearchitecting LLMs". It is the first member of a small model family built for that chapter, where a general-purpose frontier model API call is replaced by a compact, task-specific model for a narrow, well-defined use case.

Within the chapter, this model acts as the teacher for the subsequent pruning and knowledge-distillation stage that produces a smaller student model with (near-)equivalent tool-calling accuracy.

Book: Rearchitecting LLMs
Framework: OptiPFair
Technique: Supervised Fine-Tuning with LoRA (PEFT)
Chapter: Chapter 13 — Building a Specialist Model Family

Performance Metrics

The model was fine-tuned to reliably select the correct function and arguments among three tools: get_ip_zipcode, get_city_from_zipcode, and local_weather_api. Evaluation uses an exact-match metric and a tolerant-match metric that accounts for functionally equivalent values (superset city names, coordinates within 0.05°).

Base vs. Fine-Tuned (held-out test set, n=54)

Metric	Base Model	Fine-Tuned Specialist
Valid JSON	100.0%	100.0%
Exact Match	88.9%	96.3%
Tolerant Match	88.9%	96.3%
Right function, wrong args	6	2
Wrong function selected	0	0
No tool_call emitted	0	0

Overfitting Check (train sample, n=60)

Metric	Value
Exact Match	98.3%
Tolerant Match	98.3%

Train and test accuracy are close (98.3% vs. 96.3%), suggesting the model generalizes the tool-calling pattern rather than memorizing the small training set.

Note on remaining errors: Of the 2 wrong-argument cases on the test set, one involves an IP address the model was not trained to resolve correctly, and the other omits an optional num_of_days argument not always implied by the query. Both are argument-extraction edge cases, not function-selection failures.

Architecture Details

Base Model: Qwen/Qwen3-0.6B
Adapter: LoRA, rank 16, alpha 32, dropout 0.05
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable Parameters: 10,092,544 / 761,724,928 (1.33%)
Merge: LoRA adapter merged into the base weights after training (no quantization involved, so no dequantize/requantize round trip)

Training Procedure

Dataset

Source: Salesforce/xlam-function-calling-60k
Curation: Filtered to single-answer examples covering 3 domain functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api), then deduplicated by normalized query text
Curated examples: 233 → 177 after deduplication
Train/Test Split: 123 / 54, stratified by function name (test_size=0.30, random_state=42)
Loss masking: Completion-only loss — the model is trained only on the assistant's tool call, not on the system/tool-schema prompt

Hyperparameters

Epochs: 3
Batch Size: 4 (effective: 8 with gradient accumulation)
Gradient Accumulation Steps: 2
Learning Rate: 2e-4, cosine schedule, 3 warmup steps
Max Sequence Length: 1024
Precision: fp16 (AMP)

Hardware & Training Time

GPU: NVIDIA T4
Training Time: ~134s (48 steps, 3 epochs)

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oopere/qwen3-0.6b-weather-geo-specialist"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What's the weather in Barcelona?"}]
tools = [...]  # weather/geo tool schemas — see the book's companion notebook

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations & Intended Use

Intended Use

This is an educational model created as part of the Hands-on Lab in Chapter 13 of "Rearchitecting LLMs". It illustrates:

Replacing a frontier-model API call with a small, task-specific model for a narrow use case (weather/geo tool calling)
LoRA fine-tuning for function calling on a small, curated dataset
Serving as the teacher model for the pruning + knowledge-distillation stage later in the chapter

Not intended for production use. This model serves as a learning artifact within a three-function toy domain, and as a baseline for readers to compare against the distilled student model.

Limitations

Domain Coverage: Restricted to 3 functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api); no general-purpose tool-calling capability was evaluated
Training Data: Small, curated dataset (177 examples); not representative of production-scale function-calling data
Argument Extraction: Occasional errors on arguments not directly implied by the query (e.g. optional parameters, values requiring external lookups)
Language: English only

Citation

If you use this model or the techniques described in your research or projects, please cite:

Book

@book{martra2026rearchitecting,
  author    = {Pere Martra},
  title     = {Rearchitecting LLMs: Structural techniques for efficient models},
  publisher = {Manning Publications},
  year      = {2026},
  url       = {https://hubs.la/Q040tvtp0}
}

Framework

@software{optipfair2024,
  author = {Pere Martra},
  title  = {OptiPFair: Structural Pruning and Bias Analysis for LLMs},
  year   = {2024},
  url    = {https://github.com/peremartra/optipfair}
}

Acknowledgments

This model was created following the methodologies taught in "Rearchitecting LLMs" (Manning Publications, 2026). Special thanks to the Manning editorial team and the open-source community behind Hugging Face Transformers, PEFT, TRL, and PyTorch.