Llama-V-JEPA-LM-Qwen3-4B

Papers with Code: SOTA on MVP Papers with Code: SOTA on TempCompass (MCQ) Papers with Code: SOTA on Vinoground (text score) Papers with Code: SOTA on Vinoground (video score) Papers with Code: #3 on MotionBench

Paper: arXiv:2611.17831
Code and technical report: github.com/kkt94/Llama-V-JEPA-LM
Live demo: Llama-V-JEPA-LM-Llama3.1-8B in your browser (ZeroGPU)

Base language model: Qwen3-4B-Instruct-2507 (Apache-2.0). The name begins with "Llama" because the training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 whose license requires models trained on it to carry the "Llama" prefix.

Llama-V-JEPA-LM-Qwen3-4B is a video-language model from the Llama-V-JEPA-LM family. It joins a frozen V-JEPA 2.1 world-model video encoder (ViT-g) to the Qwen3-4B-Instruct-2507 language model through a spectral bridge that turns every 32-frame clip into 1,024 visual tokens. It is trained with counterfactual supervision: pixel transformations of public videos produce answer-balanced variants whose answers are fixed by construction and differ only in how the scene changes, so the language model learns to read the dynamics (direction, speed, order, state) that the world-model encoder carries. Trained on 4.53M public samples, Llama-V-JEPA-LM-Qwen3-4B reaches a 72.14 average on 12 scores across 11 video benchmarks, ahead of every open model of its size class (best open 4B-class model: Molmo2-4B, 61.08).

Highlights

  • Ahead of every open model of its size class. 72.14 on the 12-score average vs. 61.08 for Molmo2-4B; best of all open 4B-class models on 8 of 12 scores (Vinoground video 66.8 vs. 33.8, SSv2 92.2 vs. 68.3, Vinoground text 76.0 vs. 55.0, TOMATO 53.5 vs. 36.5).
  • Reads motion and temporal change. The counterfactual data teach direction, order, position, speed, counting and completion from public videos, with answers set by the transformation and no new human annotation.
  • Generalizes beyond its constructions. 96.3 average on six held-out sets (unseen speed ratios, an unseen source corpus, unseen combinations, an unseen simulator; chance 33.3), vs. 50.0 for the best open 4B-class model (Qwen3-VL-4B).
  • Compact and fast. 1,024 visual tokens per clip at any input resolution, so the language model prefills a clip with 8.6 TFLOPs in 29 ms on one H100, vs. 18.1–104.4 TFLOPs and 58–218 ms for the open models of its class; 167 ms to the first token in total.
  • Beyond Meta's V-JEPA video LLMs. On the six benchmarks reported for Meta's V-JEPA 2 / 2.1 video LLMs (Llama-3.1-8B), 69.4 average vs. 63.3 / 62.8 (as published), with 4.53M instead of 88.5M / 72.5M training samples.

Model details

Model type Video-language model (video + text in, text out)
Language model Qwen3-4B-Instruct-2507, fully fine-tuned (4.02 B parameters)
Vision encoder V-JEPA 2.1 ViT-g, frozen (1.01 B parameters)
Bridge spectral bridge: fixed PCA rotation and rescaling to 2,048 dimensions + trainable MLP (9.8 M parameters)
Total parameters 5.06 B
Visual tokens 1,024 per clip (16 tubelets Γ— 8 Γ— 8 cells), independent of input resolution
Frames 32 per clip, sampled uniformly, 384 Γ— 384
Context 2,560 tokens in training (video tokens + prompt + answer)
Precision bf16 (bridge projection statistics in fp32)
Training samples 4.53M (drawn at fixed per-source weights from a mixture of 175 public sources)
Language English
Licence Apache-2.0; V-JEPA 2 encoder: MIT; FGQA training data: Llama 3.2 Community License (see License)
Architecture details
Component Details Parameters
Video encoder (frozen) V-JEPA 2.1 ViT-g/16 (vjepa2_1_vitg_384.pt, target encoder), 384 x 384, tubelet 2, RoPE; 4 output levels (blocks 9 / 19 / 29 / 39) 1.01 B
Spectral bridge per-level LayerNorm of the 4 levels, 3 x 3 patch pooling to 8 x 8 cells per tubelet, projection onto the top 2,048 principal directions of the frozen V-JEPA features with one global scale (fixed statistics, fp32), LayerNorm + MLP 2048 -> 2048 -> 2560, plus learned position, time and cell-geometry codes 9.8 M (+ 11.54 M fp32 statistics)
Language model (fully fine-tuned) Qwen3-4B-Instruct-2507 architecture (36 layers, hidden 2560), tied input/output embeddings 4.02 B

Weights are bf16 safetensors (3 shards, about 10.1 GB). The only exception is the bridge's projection statistics (bridge.spectral.{mean,basis,scale}), which stay in fp32, also after .to(torch.bfloat16).

Quick start

1. Install (Python 3.10+, a CUDA build of PyTorch that matches your driver):

pip install torch
pip install "transformers>=5" safetensors tokenizers numpy jinja2 eva-decord huggingface_hub

2. Run on a GPU with 16 GB or more (peak memory for one clip in our test: 10.0 GiB):

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "kkt94/Llama-V-JEPA-LM-Qwen3-4B",
    trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda").eval()

# free-form answer (greedy decoding)
print(model.chat("my_video.mp4", "Describe what happens in the video."))

# multiple choice -> (index of the chosen option, one score per option)
idx, scores = model.choose("my_video.mp4", "Which way does the ball roll?", ["to the left", "to the right"])
print(idx, scores)

# only part of the video (seconds)
print(model.chat("my_video.mp4", "What does the person pick up?", start_sec=5.0, end_sec=12.0))

# encode once, ask several questions
frames, times = model.load_video("my_video.mp4")
ev = model.encode_video(frames, times)                     # [1, 1024, 2560] video tokens
print(model.choose(None, "Is the door open at the end?", ["yes", "no"], evidence=ev))
print(model.chat(None, "What happens next?", evidence=ev))

hf download kkt94/Llama-V-JEPA-LM-Qwen3-4B --local-dir ./Llama-V-JEPA-LM-Qwen3-4B downloads the files (about 10.1 GB) if you prefer a local folder; from_pretrained accepts that folder too.

Usage notes

  • choose scores the options and does not generate. The prompt is "{question}\nA. {option 0}\nB. {option 1}\n...\nAnswer with the option's letter from the given choices directly."; the prediction is the argmax of the option-letter logits at the first answer position (up to 10 options, A-J). This is the protocol of the evaluation below.
  • chat answers a single-turn question with greedy decoding (max_new_tokens, default 256).
  • Chat template (chat_template.jinja, set on model.tokenizer automatically): plain Qwen3 <|im_start|> turns, no system prompt and no thinking block. The video tokens come first in the single user turn, followed by the question. The API takes one video and one question per call.
  • Frames: 32 frames sampled uniformly over the clip (frame i at t0 + (i + 0.5) * (t1 - t0) / 32 s, nearest source frame); the clip is the whole video or [start_sec, end_sec]. Short side resized to 384 px (bicubic, antialiased), centre crop 384 x 384, ImageNet normalisation; the real frame timestamps are encoded into the video tokens. For long videos, start_sec / end_sec select the segment of interest.
  • Video files: any container / codec that decord reads (mp4, webm, mkv, avi, ...).
  • Precision: bf16, except the bridge statistics (fp32). rope_inv_freq_bf16 (config, default true) rounds the LLM's rotary frequencies to bf16 as in the evaluation these scores come from.
Repository files
File Contents
model-*.safetensors, model.safetensors.index.json weights (vision encoder, bridge, language model)
config.json JepaLlmConfig (model_type: jepa_llm) with the language model's config as text_config
configuration_jepa_llm.py the config class
modeling_jepa_llm.py JepaLlmModel: encoder, bridge, language model and the chat / choose API
processing_jepa_llm.py video decoding, frame sampling, preprocessing, prompt building, chat templates
vjepa2_1_encoder.py inference-only copy of the V-JEPA 2.1 encoder (MIT)
tokenizer files, chat_template.jinja the language model's tokenizer and the template the model was trained with
LICENSE_QWEN3, LICENSE_VJEPA2, LICENSE_LLAMA3_2, NOTICE_LLAMA3_2, USE_POLICY_LLAMA3_2.md licences (*_LLAMA3_2: Llama 3.2 licence, notice and use policy for the PLM-Video-Human FGQA training data)

The code needs no external repository (trust_remote_code=True). Tested with Python 3.12, torch 2.13, transformers 5.10.4 (see requirements.txt).

Evaluation

Twelve scores on eleven video benchmarks for Llama-V-JEPA-LM-Qwen3-4B and every open model of the 4B class. All models are evaluated by the same harness with one protocol: 32 frames, the same items and prompts, option-letter scoring. Bold: best in the table.

Model MVB PT TVB TC TOM MoB NQA MVP TB SSv2 VG-T VG-V Avg
Llama-V-JEPA-LM-Qwen3-4B 72.1 78.2 74.8 79.9 53.5 59.1 79.3 58.1 75.7 92.2 76.0 66.8 72.14
Molmo2-4B 74.2 81.4 65.5 70.8 36.5 62.3 85.1 45.4 72.8 57.0 48.2 33.8 61.08
PLM-3B 73.0 76.0 56.6 67.0 36.3 58.9 84.0 46.2 66.2 68.3 55.0 31.4 59.91
InternVL3.5-4B 70.9 69.1 60.0 71.1 32.1 59.0 79.8 41.4 73.2 64.1 53.6 28.0 58.52
Qwen3-VL-4B 68.0 72.0 59.2 70.9 36.3 61.5 80.9 39.6 68.9 64.8 47.6 31.0 58.39
Apollo-3B 62.0 67.1 50.9 63.0 29.0 55.0 80.9 27.4 60.1 64.7 37.4 25.8 51.94
Gemma 4 E4B 46.5 52.0 41.5 57.4 22.8 49.4 64.7 16.5 61.0 43.8 26.4 24.0 42.17

MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP: Minimal Video Pairs (pair accuracy), TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice), VG-T / VG-V: Vinoground text / video (pair accuracy), Avg: mean of the 12 scores.

Held-out generalization. Six construction sets built from videos and settings never used in training (unseen playback-speed ratios, a new source corpus for speed and for direction, unseen mirror Γ— reversal and acceleration Γ— reversal combinations, camera motion from an unseen simulator; chance 33.3): 96.3 average for Llama-V-JEPA-LM-Qwen3-4B, vs. 50.0 for the best open 4B-class model (Qwen3-VL-4B).

Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p clip unless noted):

Model Visual tokens Encoder (TFLOPs) LLM prefill (TFLOPs) LLM prefill (ms) First token, 480p (ms) Decode (ms / token) First token, 720p (ms)
Llama-V-JEPA-LM-Qwen3-4B 1,024 37.8 8.6 29 167 23.9 168
Qwen3-VL-4B 1,920 / 11,520 5.0 18.1 58 93 24.9 474
InternVL3.5-4B 8,192 23.2 104.4 218 273 25.3 273
Molmo2-4B 2,592 19.7 26.1 93 225 27.8 235
PLM-3B 8,192 22.2 70.4 138 228 15.6 228

First token = vision encoder + connector + LLM prefill. Decode: per generated token with KV cache. --: not timed. The visual-token count and cost of Llama-V-JEPA-LM models do not depend on the input resolution. The encoder time is a property of the encoder: V-JEPA 2.1 attends jointly over all 32 frames (9,216 patches), which is what carries the dynamics, while the open models use frame-wise image encoders (SigLIP-2-based in Qwen3-VL, InternViT, PE) that attend within one frame or frame pair at a time; it is identical for every Llama-V-JEPA-LM model on the same encoder.

Training

Data. 4.53M samples drawn at fixed per-source weights from 175 public sources: open video caption and QA collections (LLaVA-Video, ShareGPTVideo, PerceptionLM data, Charades-Ego, EPIC-Kitchens, HoloAssist, SSv2, Kinetics-700, SA-V, ActivityNet and others) plus 33 counterfactual sources (509,193 rows, about 6% of the drawn samples). The counterfactual rows come from 14 constructions in 6 families. Each takes a clip of a public dataset and a transformation; the answer follows from the transformation, so no new human annotation and no language model are involved. Within each source every scene appears with each answer equally often in expectation.

Family Constructions Rows
Sign flip clip reversed or mirrored, camera move rendered or reversed, two clips each optionally reversed 117,066
Order two events joined in sequence or stacked; which segment matches a two-action caption 85,579
Position several clips joined, one holds the queried action: beginning / middle / end 48,386
Magnitude relative speed, acceleration / deceleration, slow motion vs. time-lapse 141,084
Count one action repeated k times; number of distinct activities 35,614
Stop/cut clip cut before the action completes; physically impossible simulator edits 81,464

Recipe. The V-JEPA 2.1 encoder stays frozen. After a bridge-only warm-up (0.6–0.8% of the steps), the bridge and the full language model are trained together in one continuous stage: 30,187 steps at global batch 150, learning rate 1.2e-5 for the language model and 2e-4 for the bridge, cosine decay, weight decay 0.01 / 0.05, maximum sequence length 2,560 tokens.

Hardware. 30 NVIDIA H100 GPUs (5 clips per GPU), 16.6 hours, 499 H100-hours in total; training compute 314 EFLOPs (including the frozen encoder's forward pass).

Training data

Fine-tuned on the training splits of public video-language datasets; no new human annotation.

Dataset License Note
Something-Something v2 Qualcomm research-use license non-commercial
LLaVA-Video-178K Apache-2.0 (card: academic research) GPT-4o-generated
ShareGPTVideo (LLaVA-Hound) Apache-2.0 GPT-generated
HoloAssist CDLA-Permissive-2.0 –
SA-V CC BY 4.0 –
PLM-Video-Human (RCap, RDCap, RTLoc) CC BY 4.0 –
PLM-Video-Human FGQA Llama 3.2 Community License Llama 3.2 output
Charades / Charades-Ego AI2 non-commercial license non-commercial
EPIC-KITCHENS-100 CC BY-NC 4.0 non-commercial
EgoDex CC BY-NC-ND 4.0 non-commercial
Kinetics-700 CC BY 4.0 (annotations) YouTube videos
NVIDIA PhysicalAI Physical-Interaction-Scenes OpenMDW-1.1 –
RoboVQA CC BY 4.0 –
Language-Table Apache-2.0 –
CLEVRER see project page –
TimeIT CC BY 4.0 (instructions) source videos under their own terms
PerceptionTest (train) CC BY 4.0 –
STAR (train) Apache-2.0 Charades videos
Oops! CC BY-NC-SA 4.0 non-commercial
HT-Step CC BY-NC 4.0 non-commercial
Others (ActivityNet, YouCook2, LSMDC, Ego-QA, Video-Holmes, VidLN, Countix) see each dataset research use
UltraChat-200k prompts MIT answers generated by the base LLM
Reasoning / answer rewrites generated by Qwen3-8B / Qwen3-VL-8B (Apache-2.0) –

Each dataset remains under its own terms; this release grants no rights in them.

Intended use

  • Intended: question answering about and description of videos in English: actions, events and their order, motion direction and speed, camera motion, counting, and state changes; research on video-language models and world-model encoders.
  • Out of scope: image, document, chart or OCR question answering, which were not training targets; use that violates the Llama 3.2 Acceptable Use Policy (which applies to the FGQA training-data component) or applicable law.

Several training datasets restrict commercial use. We release the weights for research and non-commercial use; anyone considering commercial use is responsible for checking the terms of the training data.

Limitations

  • Single images and text in frames. V-JEPA 2.1 is trained only on video, to predict masked spatiotemporal regions in latent space. It is never trained to align with text or to read it. Text recognition (OCR, documents, charts, slides) and single-image question answering are therefore outside what the encoder was built for, and they are not the target of this model. It is built for how scenes change over time. For text-heavy or single-image tasks, an image–text encoder is the natural choice.
  • Long videos. The model targets clip-level understanding of dynamics, 32 frames per clip. Hour-scale long-video benchmarks need many clips per video and far longer contexts, in training as well as at inference. At the scale of this release (1B–30B, one training stage of a few hundred H100-hours), that training cost is impractical, so we did not train or evaluate for it. For a long video, start_sec / end_sec select the segment to ask about.

License

This model combines components under the following licences:

  • Language model: Apache License 2.0. The language-model weights are a modified version of Qwen3-4B-Instruct-2507 (LICENSE_QWEN3): all weights were fine-tuned on video-language data, and video tokens from the bridge are inserted into its input embeddings. The tokenizer files are unmodified copies from Qwen3-4B-Instruct-2507.
  • Video encoder: MIT License. The encoder weights are the V-JEPA 2.1 ViT-g checkpoint vjepa2_1_vitg_384.pt released by Meta with facebookresearch/vjepa2 (MIT, LICENSE_VJEPA2), unchanged (frozen). vjepa2_1_encoder.py is adapted from that repository under the MIT License.
  • Bridge weights and model code are released under the Apache License 2.0.
  • Training data component: Llama 3.2 Community License. The training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 that is subject to the Llama 3.2 Community License (LICENSE_LLAMA3_2) and the Llama 3.2 Acceptable Use Policy (USE_POLICY_LLAMA3_2.md); as that license requires for models trained on it, the model name begins with "Llama". Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright Β© Meta Platforms, Inc. All Rights Reserved. (NOTICE_LLAMA3_2)
  • Training datasets remain under their own terms (see Training data).

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government (MSIT) (No. RS-2026-25505492, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale).

Citation

@misc{kim2026vjepalm,
  title         = {V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations},
  author        = {Kim, Keuntae and Kim, Dongjun and Choi, Yong Suk},
  year          = {2026},
  eprint        = {2611.17831},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.17831}
}

Please also cite V-JEPA 2 / V-JEPA 2.1 and Qwen3.

Model family

All sizes are in the Llama-V-JEPA-LM collection.

Model Language model Encoder Avg (12)
Llama-V-JEPA-LM-Llama3.2-1B Llama-3.2-1B-Instruct V-JEPA 2.1 ViT-L 62.21
Llama-V-JEPA-LM-Qwen3-1.7B Qwen3-1.7B V-JEPA 2.1 ViT-L 66.02
Llama-V-JEPA-LM-Llama3.2-3B Llama-3.2-3B-Instruct V-JEPA 2.1 ViT-g 68.97
Llama-V-JEPA-LM-Qwen3-4B (this model) Qwen3-4B-Instruct-2507 V-JEPA 2.1 ViT-g 72.14
Llama-V-JEPA-LM-Llama3.1-8B Llama-3.1-8B-Instruct V-JEPA 2.1 ViT-g 70.51
Llama-V-JEPA-LM-Qwen3-8B Qwen3-8B V-JEPA 2.1 ViT-g 70.11
Llama-V-JEPA-LM-Qwen3-30B-A3B Qwen3-30B-A3B V-JEPA 2.1 ViT-G 69.49

Model card authors

Keuntae Kim, Hanyang University, Artificial Intelligence Lab

Downloads last month
303
Safetensors
Model size
5B params
Tensor type
F32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for kkt94/Llama-V-JEPA-LM-Qwen3-4B

Finetuned
(2,769)
this model

Space using kkt94/Llama-V-JEPA-LM-Qwen3-4B 1

Collection including kkt94/Llama-V-JEPA-LM-Qwen3-4B

Paper for kkt94/Llama-V-JEPA-LM-Qwen3-4B

Evaluation results