baguettotron-vision-vqa

v1.1 — 4 November 2026: weight-space edit. 9,216 weights changed by hand, no training: the model now ends its answers on a real stop token and answers "Is there a … in the image?" better. See v1.1: weight-space edit below. The original release is kept as revision="v1.0".

Baguettotron-VLM is an open, fully-reproducible, multilingual Vision-Language Model in the sub-1B parameter class. It extends PleIAs/Baguettotron — a 321M text-only reasoning SLM — with visual capabilities via the InternViT-300M-448px-V2.5 vision encoder and a lightweight MLP projector, for a total of ~628M parameters. It inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone.

Two checkpoints are published:

baguettotron-internvit-alignment — the projector-only warmup. It describes images, and nothing more.
baguettotron-vision-vqa — instruction-tuned on top of it. It goes past plain description and follows visual instructions, so prefer it for a richer chat experience.

Apache 2.0. Both live in the Baguettotron-VLM collection. Source: github.com/andreagemelli/baguettotron-vlm · Write-up: andreagemelli.me/posts/baguettotron-vlm

Instruction-tuned checkpoint. Full fine-tuning of the projector + LLM on The Cauldron (47 subsets) mixed with 10% PleIAs/SYNTH text-only data. It follows visual instructions and answers VQA directly, without <think> traces. It is at its best on short descriptions and "what is in the picture" questions, and weakest on long open description; multiple-choice, counting and colour answers are close to chance.

Architecture

Image (448×448)
  → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d
  → Pixel unshuffle (factor=2)                → 256 tokens × 4096d
  → MLP projector (2-layer, ~2.7M)            → 256 tokens × 576d
  → Interleave with text tokens
  → Baguettotron (321M, Llama arch, 80L, h=576)

Total: ~628M parameters

Usage

pip install "transformers>=4.56,<5" torch pillow timm einops accelerate

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "andreagemelli/baguettotron-vision-vqa",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",  # also tested on Apple Silicon (mps) and CPU
)
processor = AutoProcessor.from_pretrained(
    "andreagemelli/baguettotron-vision-vqa",
    trust_remote_code=True,
)

image = Image.open("photo.jpg").convert("RGB")
inputs = processor(
    messages=[{"role": "user", "content": "<image>\nDescribe the image concisely."}],
    image=image,
)
inputs = {k: v.to(model.device) for k, v in inputs.items() if v is not None}

print(model.chat(**inputs))

Example output

The snippet above as written (greedy, bf16) on Apple Silicon, prompt Describe the image concisely. — verbatim output. Images from COCO val2017.

	v1.1	v1.0
	`In this image, two cats are sitting on a purple carpet. One cat is wearing a red light switch and the other has a white cape`	`In this image, two cats are sitting on a purple carpet. One cat is wearing a red light switch and the other has a button.`
	`The image shows a bear sitting on the grass`	`The image shows a bear sitting on the grass.`
	`A stop sign on the side of a road. The sky is blue`	`A stop sign on the side of a road. The sky is blue.`
	`The red bus is on the road`	`The red bus is on the road.`

bf16 descriptions vary with the device. In fp32 both versions write the same descriptions apart from the final period; for the cats, The image is a picture of two cats sitting on the couch. The cat has its head down and legs are raised.

Chat template

Trained with the assistant turn always starting with a closing </think> token ("no-thinking" mode) followed by the answer, because The Cauldron carries no reasoning traces. The processor pre-fills that </think>\n prefix after <|im_start|>assistant\n when add_generation_prompt=True (the default), so you pass the user message and the model completes the answer.

For the raw prompt without the auto-prefix, pass add_generation_prompt=False to the processor.

v1.1: weight-space edit

Inspired by Sparks of Weight-Space Programming (Takehi, Teoh, Zhang, Khattab, 2026): change a trained network by reading its activations and writing its weights directly, without gradient descent. Their rule held here too: an edit can unlock what a model already knows, not teach it something new. This network already encodes whether the object asked about is in the image (a linear probe on its activations reads it 70–74% of the time, and is at chance with a mismatched image), but its Yes/No answer ignored that evidence. v1.1 routes it to the answer.

Weights touched: 9,216 of 665M (0.0014%). 4 rows of the output layer (the stop token, 2,304 weights) and 4 neurons of the last layer's MLP: 2 for the presence circuit and 2 for a loop breaker that ends a long answer once it starts repeating a sentence (3,456 weights per pair). The file stores the output layer apart from the input embeddings, hence 665M rather than the ~628M above.

On images and questions never used to derive the edits (the model's own greedy answer, fp32):

test	v1.0	v1.1
POPE random — 600 "Is there a … in the image?" on 100 unseen COCO images (chance: 50%)	52.3%	64.0%
POPE random, asked without the answer-format suffix	48.3%	69.3%
POPE adversarial	50.0%	54.5%
POPE adversarial, without the suffix	49.3%	56.0%
"Is there a cat in the picture?" on the cats example	`No.`	`Yes`
long descriptions that end by themselves (plain `generate()`, no repetition penalty)	0%	83%
RealWorldQA, exact match (379 unseen rows)	21.6%	21.9%
VQAv2 val (1,780 questions)	33.5	33.7
GQA testdev-balanced, yes/no questions	53.7%	50.7%

In bf16, the dtype in Usage, POPE random goes 54.5 → 63.0% (without the suffix 49.3 → 66.7%) and adversarial 52.8 → 54.5% (50.7 → 56.0%).

Stop token. Training masked <|im_end|> out of the loss, so v1.0 never emits it: it spells the marker out as text (.< | im _ end |>) and chat() cuts it with a text rule. v1.1 swaps 4 output rows so those pieces land on the real stop ids. The swap changes nothing else: greedy answers are the old ones, minus the final "." that the stop token now consumes. Plain generate() stops by itself (56 of 60 RealWorldQA answers, against 0 for v1.0), and short answers come back 20–45% faster.
Known cost. The presence gate also fires on other yes/no questions ("Is the car red?"), where the model holds no evidence. On GQA that costs 3 points of yes/no accuracy; VQAv2 is unchanged.

To load the original release, pass revision="v1.0" to both from_pretrained calls. More in the write-up update.

Limitations

Resolution ceiling. One 448×448 crop → 256 visual tokens puts document text at roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not by budget. Neither published checkpoint reads text in an image.
Hallucinations, especially on fine-grained or text-heavy questions. Asking for more than a short answer it starts inventing some funny stuff - so be aware when using it thay may produce some hallucinated content.
Would benefit from more instruction tuning. The visual instruction budget is small relative to the size of the data mixture, and it shows on long or compositional prompts.
Multilingual capability is inherited, not verified. The backbone covers six languages; the VLM was never evaluated on non-English benchmarks.

Tested against transformers 4.57. Newer major versions may need adjustments.

Contributions and suggestions are very welcome — issues, PRs, and ideas for better data mixes, training recipes, or evaluation setups are all appreciated. Open an issue or PR on the GitHub repo.

Citation

If you use or extend Baguettotron-VLM in your research, please cite it:

@misc{gemelli2026baguettotronvlm,
  title        = {Baguettotron-VLM: An Open, Reproducible, Multilingual Sub-1B Vision-Language Model},
  author       = {Gemelli, Andrea},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/andreagemelli/baguettotron-vision-vqa}},
  note         = {Code: \url{https://github.com/andreagemelli/baguettotron-vlm}}
}