# VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

arXiv: https://arxiv.org/abs/2512.10942 · HTML: https://mlsanity.com/paper/2512.10942/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 74 of its 90 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 74)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)

## Built on by (most cited first, 30 of 68)

- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models](https://mlsanity.com/paper/2602.01166/) (2026)
- [VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models](https://mlsanity.com/paper/2601.14354/) (2026)
- [Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models](https://mlsanity.com/paper/2603.21389/) (2026)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model](https://mlsanity.com/paper/2603.22281/) (2026)
- [Action100M: A Large-scale Video Action Dataset](https://mlsanity.com/paper/2601.10592/) (2026)
- [Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding](https://mlsanity.com/paper/2603.19235/) (2026)
- [CGM-JEPA: Learning Consistent Continuous Glucose Monitor Representations via Predictive Self-Supervised Pretraining](https://mlsanity.com/paper/2605.00933/) (2026)
- [Orca: The World is in Your Mind](https://mlsanity.com/paper/2606.30534/) (2026)
- [A Generalization Theory for JEPA-Based World Models](https://mlsanity.com/paper/2606.27014/) (2026)
- [Recursive Belief Vision Language Action Models](https://mlsanity.com/paper/2602.20659/) (2026)
- [Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction](https://mlsanity.com/paper/2603.16281/) (2026)
- [Multimodal Latent Reasoning via Predictive Embeddings](https://mlsanity.com/paper/2604.08065/) (2026)
- [AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modeling](https://mlsanity.com/paper/2605.05586/) (2026)
- [Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement](https://mlsanity.com/paper/2605.14759/) (2026)
- [RISE: Adaptive Imagination for World Action Models](https://mlsanity.com/paper/2608.20430/) (2026)
- [The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR](https://mlsanity.com/paper/2601.22128/) (2026)
- [Self-Distillation of Hidden Layers for Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.15553/) (2026)
- [CSI-JEPA: Towards Foundation Representations for Ubiquitous Sensing with Minimal Supervision](https://mlsanity.com/paper/2605.14171/) (2026)
- [World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications](https://mlsanity.com/paper/2606.00133/) (2026)
- [Gaussian Joint Embeddings For Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.26799/) (2026)
- [From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning](https://mlsanity.com/paper/2604.13518/) (2026)
- [Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories](https://mlsanity.com/paper/2605.10840/) (2026)
- [LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives](https://mlsanity.com/paper/2607.00784/) (2026)
- [Unlocked Backpropagation using Wave Scattering](https://mlsanity.com/paper/2602.10461/) (2026)
- [TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models](https://mlsanity.com/paper/2603.06999/) (2026)
- [PVI: Plug-in Visual Injection for Vision-Language-Action Models](https://mlsanity.com/paper/2603.12772/) (2026)

## Abstract (arXiv)

We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.