citiusLTL/gemma-3-4b-it-who-sft-qlora

Gemma 3 4B-IT WHO-SFT QLoRA Adapter

This repository contains a QLoRA adapter for unsloth/gemma-3-4b-it-unsloth-bnb-4bit. It was created as part of the MSc thesis Construction of Medical Question-Answering Training Collections and Fine-Tuning of Large Language Models for Reliable Medical Question Answering.

Training corpus

The adapter was trained with WHO-SFT, a collection of 4,057 English medical question-answer pairs generated from 237 WHO Fact Sheets. Questions and answers were generated from source text in chunks, structurally validated, deduplicated, and converted to the same supervised fine-tuning format as Drugs-SFT.

Code, data-preparation scripts, experiment definitions, and evaluation outputs are available at:

https://github.com/braisglopez/medical-qa-sft

Training configuration

• Base model: Gemma 3 4B-IT quantized to 4 bits.
• Adaptation method: QLoRA.
• LoRA rank: 8.
• LoRA alpha: 8.
• Learning rate: 5e-5.
• Epochs: 1.
• Maximum sequence length: 1,024 tokens.

Evaluation

Evaluation used the full MIRAGE benchmark with constrained selection among the valid answer options for each question.

Subset	Base model	WHO-SFT adapter
MedQA	47.45%	44.30%
MedMCQA	44.66%	43.01%
PubMedQA	48.80%	56.00%
BioASQ	70.71%	66.83%
Medical MMLU	57.12%	54.45%
Overall	49.26%	47.62%

Loading the adapter

This repository contains adapter weights only. Load the compatible base model first and apply this adapter with PEFT.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "unsloth/gemma-3-4b-it-unsloth-bnb-4bit"
adapter_id = "citiusLTL/gemma-3-4b-it-who-sft-qlora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

Limitations and intended use

This adapter is intended solely for research. It is not a clinical device and must not be used for diagnosis, treatment, or medical decision-making. MIRAGE evaluates constrained answers to closed questions; its results do not constitute clinical validation.