Llama-V-JEPA-LM-Llama3.1-8B

Paper: arXiv:2611.17831
Code and technical report: github.com/kkt94/Llama-V-JEPA-LM
Live demo: try this model in your browser (ZeroGPU)

Built with Llama

Llama-V-JEPA-LM-Llama3.1-8B is built on Meta's Llama-3.1-8B-Instruct. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright Β© Meta Platforms, Inc. All Rights Reserved. Use of this model is subject to that licence and to the Llama 3.1 Acceptable Use Policy.

The training data also include the FGQA split of PLM-Video-Human, an output of Llama 3.2; that training-data component is subject to the Llama 3.2 Community License (LICENSE_LLAMA3_2, NOTICE_LLAMA3_2, USE_POLICY_LLAMA3_2.md), which requires models trained on it to carry the "Llama" prefix.

Llama-V-JEPA-LM-Llama3.1-8B is a video-language model from the Llama-V-JEPA-LM family. It joins a frozen V-JEPA 2.1 world-model video encoder (ViT-g) to the Llama-3.1-8B-Instruct language model through a spectral bridge that turns every 32-frame clip into 1,024 visual tokens. It is trained with counterfactual supervision: pixel transformations of public videos produce answer-balanced variants whose answers are fixed by construction and differ only in how the scene changes, so the language model learns to read the dynamics (direction, speed, order, state) that the world-model encoder carries. Trained on 4.53M public samples, Llama-V-JEPA-LM-Llama3.1-8B reaches a 70.51 average on 12 scores across 11 video benchmarks, ahead of every open model of its size class (best open 8B-class model: PLM-8B, 64.79).

Highlights

  • Ahead of every open model of its size class. 70.51 on the 12-score average vs. 64.79 for PLM-8B; best of all open 8B-class models on 8 of 12 scores (SSv2 92.3 vs. 67.9, Vinoground video 63.0 vs. 44.8, TOMATO 55.5 vs. 42.6, Vinoground text 74.2 vs. 63.4).
  • Reads motion and temporal change. The counterfactual data teach direction, order, position, speed, counting and completion from public videos, with answers set by the transformation and no new human annotation.
  • Generalizes beyond its constructions. 96.1 average on six held-out sets (unseen speed ratios, an unseen source corpus, unseen combinations, an unseen simulator; chance 33.3), vs. 55.5 for the best open 8B-class model (Qwen3-VL-8B).
  • Compact and fast. 1,024 visual tokens per clip at any input resolution, so the language model prefills a clip with 16.2 TFLOPs in 36 ms on one H100, vs. 32.2–160.7 TFLOPs and 81–314 ms for the open models of its class; 175 ms to the first token in total.
  • Beyond Meta's V-JEPA video LLMs. On the six benchmarks reported for Meta's V-JEPA 2 / 2.1 video LLMs (Llama-3.1-8B), 68.2 average vs. 63.3 / 62.8 (as published), with 4.53M instead of 88.5M / 72.5M training samples.

Model details

Model type Video-language model (video + text in, text out)
Language model Llama-3.1-8B-Instruct, fully fine-tuned (8.03 B parameters)
Vision encoder V-JEPA 2.1 ViT-g, frozen (1.01 B parameters)
Bridge spectral bridge: fixed PCA rotation and rescaling to 2,048 dimensions + trainable MLP (13.2 M parameters)
Total parameters 9.07 B
Visual tokens 1,024 per clip (16 tubelets Γ— 8 Γ— 8 cells), independent of input resolution
Frames 32 per clip, sampled uniformly, 384 Γ— 384
Context 2,560 tokens in training (video tokens + prompt + answer)
Precision bf16 (bridge projection statistics in fp32)
Training samples 4.53M (drawn at fixed per-source weights from a mixture of 175 public sources)
Language English
Licence Llama 3.1 Community License; V-JEPA 2 encoder: MIT; FGQA training data: Llama 3.2 Community License (see License)
Architecture details
Component Details Parameters
Video encoder (frozen) V-JEPA 2.1 ViT-g/16 (vjepa2_1_vitg_384.pt, target encoder), 384 x 384, tubelet 2, RoPE; 4 output levels (blocks 9 / 19 / 29 / 39) 1.01 B
Spectral bridge per-level LayerNorm of the 4 levels, 3 x 3 patch pooling to 8 x 8 cells per tubelet, projection onto the top 2,048 principal directions of the frozen V-JEPA features with one global scale (fixed statistics, fp32), LayerNorm + MLP 2048 -> 2048 -> 4096, plus learned position, time and cell-geometry codes 13.2 M (+ 11.54 M fp32 statistics)
Language model (fully fine-tuned) Llama-3.1-8B-Instruct architecture (32 layers, hidden 4096), separate output head (untied embeddings) 8.03 B

Weights are bf16 safetensors (5 shards, about 18.2 GB). The only exception is the bridge's projection statistics (bridge.spectral.{mean,basis,scale}), which stay in fp32, also after .to(torch.bfloat16).

Quick start

1. Install (Python 3.10+, a CUDA build of PyTorch that matches your driver):

pip install torch
pip install "transformers>=5" safetensors tokenizers numpy jinja2 eva-decord huggingface_hub

2. Run on a GPU with 24 GB or more (peak memory for one clip in our test: 17.5 GiB):

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "kkt94/Llama-V-JEPA-LM-Llama3.1-8B",
    trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda").eval()

# free-form answer (greedy decoding)
print(model.chat("my_video.mp4", "Describe what happens in the video."))

# multiple choice -> (index of the chosen option, one score per option)
idx, scores = model.choose("my_video.mp4", "Which way does the ball roll?", ["to the left", "to the right"])
print(idx, scores)

# only part of the video (seconds)
print(model.chat("my_video.mp4", "What does the person pick up?", start_sec=5.0, end_sec=12.0))

# encode once, ask several questions
frames, times = model.load_video("my_video.mp4")
ev = model.encode_video(frames, times)                     # [1, 1024, 4096] video tokens
print(model.choose(None, "Is the door open at the end?", ["yes", "no"], evidence=ev))
print(model.chat(None, "What happens next?", evidence=ev))

hf download kkt94/Llama-V-JEPA-LM-Llama3.1-8B --local-dir ./Llama-V-JEPA-LM-Llama3.1-8B downloads the files (about 18.2 GB) if you prefer a local folder; from_pretrained accepts that folder too.

Usage notes

  • choose scores the options and does not generate. The prompt is "{question}\nA. {option 0}\nB. {option 1}\n...\nAnswer with the option's letter from the given choices directly."; the prediction is the argmax of the option-letter logits at the first answer position (up to 10 options, A-J). This is the protocol of the evaluation below.
  • chat answers a single-turn question with greedy decoding (max_new_tokens, default 256).
  • Chat template (chat_template.jinja, set on model.tokenizer automatically): the Llama 3 Instruct template with its system header and no system text; the header's date line is fixed to Today Date: 26 Jul 2024 (the template default), exactly as in training and evaluation. The video tokens come first in the single user turn, followed by the question. The API takes one video and one question per call.
  • Frames: 32 frames sampled uniformly over the clip (frame i at t0 + (i + 0.5) * (t1 - t0) / 32 s, nearest source frame); the clip is the whole video or [start_sec, end_sec]. Short side resized to 384 px (bicubic, antialiased), centre crop 384 x 384, ImageNet normalisation; the real frame timestamps are encoded into the video tokens. For long videos, start_sec / end_sec select the segment of interest.
  • Video files: any container / codec that decord reads (mp4, webm, mkv, avi, ...).
  • Precision: bf16, except the bridge statistics (fp32). rope_inv_freq_bf16 (config, default true) rounds the LLM's rotary frequencies to bf16 as in the evaluation these scores come from.
Repository files
File Contents
model-*.safetensors, model.safetensors.index.json weights (vision encoder, bridge, language model)
config.json JepaLlmConfig (model_type: jepa_llm) with the language model's config as text_config
configuration_jepa_llm.py the config class
modeling_jepa_llm.py JepaLlmModel: encoder, bridge, language model and the chat / choose API
processing_jepa_llm.py video decoding, frame sampling, preprocessing, prompt building, chat templates
vjepa2_1_encoder.py inference-only copy of the V-JEPA 2.1 encoder (MIT)
tokenizer files, chat_template.jinja the language model's tokenizer and the template the model was trained with
LICENSE (Llama licence), USE_POLICY.md, NOTICE, LICENSE_VJEPA2, LICENSE_LLAMA3_2, NOTICE_LLAMA3_2, USE_POLICY_LLAMA3_2.md licences and the Llama attribution notice (*_LLAMA3_2: Llama 3.2 licence, notice and use policy for the PLM-Video-Human FGQA training data)

The code needs no external repository (trust_remote_code=True). Tested with Python 3.12, torch 2.13, transformers 5.10.4 (see requirements.txt).

Evaluation

Twelve scores on eleven video benchmarks for Llama-V-JEPA-LM-Llama3.1-8B and every open model of the 8B class. All models are evaluated by the same harness with one protocol: 32 frames, the same items and prompts, option-letter scoring. Bold: best in the table.

Model MVB PT TVB TC TOM MoB NQA MVP TB SSv2 VG-T VG-V Avg
Llama-V-JEPA-LM-Llama3.1-8B 71.9 75.0 71.8 77.6 55.5 55.9 79.0 57.6 72.3 92.3 74.2 63.0 70.51
PLM-8B 76.0 80.6 62.7 70.9 42.6 62.7 85.7 49.0 71.2 67.9 63.4 44.8 64.79
Molmo2-8B 75.6 81.9 64.1 72.1 35.4 61.4 85.4 43.4 70.8 55.2 49.2 33.8 60.69
Qwen3-VL-8B 68.8 73.0 58.2 73.6 36.5 62.9 82.1 41.8 70.8 65.9 51.0 35.0 59.97
InternVL3.5-8B 71.0 72.7 62.7 71.0 33.6 58.1 81.5 38.8 72.1 66.7 47.2 28.4 58.65
VideoLLaMA3-7B 65.6 69.7 49.2 69.3 29.4 57.8 81.5 29.9 70.5 59.1 44.8 28.6 54.62

MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP: Minimal Video Pairs (pair accuracy), TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice), VG-T / VG-V: Vinoground text / video (pair accuracy), Avg: mean of the 12 scores.

Held-out generalization. Six construction sets built from videos and settings never used in training (unseen playback-speed ratios, a new source corpus for speed and for direction, unseen mirror Γ— reversal and acceleration Γ— reversal combinations, camera motion from an unseen simulator; chance 33.3): 96.1 average for Llama-V-JEPA-LM-Llama3.1-8B, vs. 55.5 for the best open 8B-class model (Qwen3-VL-8B).

Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p clip unless noted):

Model Visual tokens Encoder (TFLOPs) LLM prefill (TFLOPs) LLM prefill (ms) First token, 480p (ms) Decode (ms / token) First token, 720p (ms)
Llama-V-JEPA-LM-Llama3.1-8B 1,024 37.8 16.2 36 175 17.1 175
Qwen3-VL-8B 1,920 / 11,520 6.8 32.2 81 124 25.2 666
InternVL3.5-8B 8,192 23.2 160.7 314 370 24.5 369
Molmo2-8B 2,592 19.7 45.4 127 261 28.4 271
PLM-8B 8,192 123.6 151.9 270 610 18.3 610

First token = vision encoder + connector + LLM prefill. Decode: per generated token with KV cache. --: not timed. The visual-token count and cost of Llama-V-JEPA-LM models do not depend on the input resolution. The encoder time is a property of the encoder: V-JEPA 2.1 attends jointly over all 32 frames (9,216 patches), which is what carries the dynamics, while the open models use frame-wise image encoders (SigLIP-2-based in Qwen3-VL, InternViT, PE) that attend within one frame or frame pair at a time; it is identical for every Llama-V-JEPA-LM model on the same encoder.

Training

Data. 4.53M samples drawn at fixed per-source weights from 175 public sources: open video caption and QA collections (LLaVA-Video, ShareGPTVideo, PerceptionLM data, Charades-Ego, EPIC-Kitchens, HoloAssist, SSv2, Kinetics-700, SA-V, ActivityNet and others) plus 33 counterfactual sources (509,193 rows, about 6% of the drawn samples). The counterfactual rows come from 14 constructions in 6 families. Each takes a clip of a public dataset and a transformation; the answer follows from the transformation, so no new human annotation and no language model are involved. Within each source every scene appears with each answer equally often in expectation.

Family Constructions Rows
Sign flip clip reversed or mirrored, camera move rendered or reversed, two clips each optionally reversed 117,066
Order two events joined in sequence or stacked; which segment matches a two-action caption 85,579
Position several clips joined, one holds the queried action: beginning / middle / end 48,386
Magnitude relative speed, acceleration / deceleration, slow motion vs. time-lapse 141,084
Count one action repeated k times; number of distinct activities 35,614
Stop/cut clip cut before the action completes; physically impossible simulator edits 81,464

Recipe. The V-JEPA 2.1 encoder stays frozen. After a bridge-only warm-up (0.6–0.8% of the steps), the bridge and the full language model are trained together in one continuous stage: 30,187 steps at global batch 150, learning rate 8e-6 for the language model and 2e-4 for the bridge, cosine decay, weight decay 0.01 / 0.05, maximum sequence length 2,560 tokens.

Hardware. 30 NVIDIA H100 GPUs (5 clips per GPU), 19.9 hours, 596 H100-hours in total; training compute 426 EFLOPs (including the frozen encoder's forward pass).

Training data

Fine-tuned on the training splits of public video-language datasets; no new human annotation.

Dataset License Note
Something-Something v2 Qualcomm research-use license non-commercial
LLaVA-Video-178K Apache-2.0 (card: academic research) GPT-4o-generated
ShareGPTVideo (LLaVA-Hound) Apache-2.0 GPT-generated
HoloAssist CDLA-Permissive-2.0 –
SA-V CC BY 4.0 –
PLM-Video-Human (RCap, RDCap, RTLoc) CC BY 4.0 –
PLM-Video-Human FGQA Llama 3.2 Community License Llama 3.2 output
Charades / Charades-Ego AI2 non-commercial license non-commercial
EPIC-KITCHENS-100 CC BY-NC 4.0 non-commercial
EgoDex CC BY-NC-ND 4.0 non-commercial
Kinetics-700 CC BY 4.0 (annotations) YouTube videos
NVIDIA PhysicalAI Physical-Interaction-Scenes OpenMDW-1.1 –
RoboVQA CC BY 4.0 –
Language-Table Apache-2.0 –
CLEVRER see project page –
TimeIT CC BY 4.0 (instructions) source videos under their own terms
PerceptionTest (train) CC BY 4.0 –
STAR (train) Apache-2.0 Charades videos
Oops! CC BY-NC-SA 4.0 non-commercial
HT-Step CC BY-NC 4.0 non-commercial
Others (ActivityNet, YouCook2, LSMDC, Ego-QA, Video-Holmes, VidLN, Countix) see each dataset research use
UltraChat-200k prompts MIT answers generated by the base LLM
Reasoning / answer rewrites generated by Qwen3-8B / Qwen3-VL-8B (Apache-2.0) –

Each dataset remains under its own terms; this release grants no rights in them.

Intended use

  • Intended: question answering about and description of videos in English: actions, events and their order, motion direction and speed, camera motion, counting, and state changes; research on video-language models and world-model encoders.
  • Out of scope: image, document, chart or OCR question answering, which were not training targets; use that violates the Llama 3.1 Acceptable Use Policy or the Llama 3.2 Acceptable Use Policy or applicable law.

Several training datasets restrict commercial use. We release the weights for research and non-commercial use; anyone considering commercial use is responsible for checking the terms of the training data.

Limitations

  • Single images and text in frames. V-JEPA 2.1 is trained only on video, to predict masked spatiotemporal regions in latent space. It is never trained to align with text or to read it. Text recognition (OCR, documents, charts, slides) and single-image question answering are therefore outside what the encoder was built for, and they are not the target of this model. It is built for how scenes change over time. For text-heavy or single-image tasks, an image–text encoder is the natural choice.
  • Long videos. The model targets clip-level understanding of dynamics, 32 frames per clip. Hour-scale long-video benchmarks need many clips per video and far longer contexts, in training as well as at inference. At the scale of this release (1B–30B, one training stage of a few hundred H100-hours), that training cost is impractical, so we did not train or evaluate for it. For a long video, start_sec / end_sec select the segment to ask about.

License

This model combines components under the following licences:

  • Language model: Llama 3.1 Community License. The language-model weights and tokenizer are a fine-tuned derivative of Llama-3.1-8B-Instruct (Llama Materials), governed by the Llama 3.1 Community License (LICENSE) and the Llama 3.1 Acceptable Use Policy (USE_POLICY.md). Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright Β© Meta Platforms, Inc. All Rights Reserved. (NOTICE)
  • Video encoder: MIT License. The encoder weights are the V-JEPA 2.1 ViT-g checkpoint vjepa2_1_vitg_384.pt released by Meta with facebookresearch/vjepa2 (MIT, LICENSE_VJEPA2), unchanged (frozen). vjepa2_1_encoder.py is adapted from that repository under the MIT License.
  • Bridge weights and model code are distributed together with, and subject to, the Llama 3.1 Community License as part of this Llama derivative.
  • Training data component: Llama 3.2 Community License. The training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 that is subject to the Llama 3.2 Community License (LICENSE_LLAMA3_2) and the Llama 3.2 Acceptable Use Policy (USE_POLICY_LLAMA3_2.md); as that license requires for models trained on it, the model name begins with "Llama". Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright Β© Meta Platforms, Inc. All Rights Reserved. (NOTICE_LLAMA3_2)
  • Training datasets remain under their own terms (see Training data).

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government (MSIT) (No. RS-2026-25505492, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale).

Citation

@misc{kim2026vjepalm,
  title         = {V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations},
  author        = {Kim, Keuntae and Kim, Dongjun and Choi, Yong Suk},
  year          = {2026},
  eprint        = {2611.17831},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.17831}
}

Please also cite V-JEPA 2 / V-JEPA 2.1 and Llama 3.

Model family

All sizes are in the Llama-V-JEPA-LM collection.

Model Language model Encoder Avg (12)
Llama-V-JEPA-LM-Llama3.2-1B Llama-3.2-1B-Instruct V-JEPA 2.1 ViT-L 62.21
Llama-V-JEPA-LM-Qwen3-1.7B Qwen3-1.7B V-JEPA 2.1 ViT-L 66.02
Llama-V-JEPA-LM-Llama3.2-3B Llama-3.2-3B-Instruct V-JEPA 2.1 ViT-g 68.97
Llama-V-JEPA-LM-Qwen3-4B Qwen3-4B-Instruct-2507 V-JEPA 2.1 ViT-g 72.14
Llama-V-JEPA-LM-Llama3.1-8B (this model) Llama-3.1-8B-Instruct V-JEPA 2.1 ViT-g 70.51
Llama-V-JEPA-LM-Qwen3-8B Qwen3-8B V-JEPA 2.1 ViT-g 70.11
Llama-V-JEPA-LM-Qwen3-30B-A3B Qwen3-30B-A3B V-JEPA 2.1 ViT-G 69.49

Model card authors

Keuntae Kim, Hanyang University, Artificial Intelligence Lab

Downloads last month
681
Safetensors
Model size
9B params
Tensor type
F32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for kkt94/Llama-V-JEPA-LM-Llama3.1-8B

Finetuned
(3,324)
this model

Space using kkt94/Llama-V-JEPA-LM-Llama3.1-8B 1

Collection including kkt94/Llama-V-JEPA-LM-Llama3.1-8B

Paper for kkt94/Llama-V-JEPA-LM-Llama3.1-8B

Evaluation results