mlsanityData to 7 Oct 2026

VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

arXiv 2512.10942 ↗First posted 11 Dec 2025cs.CVcited by 61

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 90 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

74 of 90 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
  2. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  3. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations17 Jun
  4. 2024An Introduction to Vision-Language Modeling27 May
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  11. 2017Localizing Moments in Video with Natural Languagecited by 660
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  13. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  15. 2020Rescaling Egocentric Visioncited by 529
  16. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  17. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  18. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  19. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  20. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  21. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  22. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  23. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  24. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  25. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  26. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  27. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  28. 2018TallyQA: Answering Complex Counting Questionscited by 152
  29. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  30. 2022Linearly Mapping from Image to Text Spacecited by 139
  31. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  32. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  33. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  34. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  35. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  36. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  37. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  38. 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understandingcited by 88

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 61.

61 papers.

Sort:

By area:

  1. 2026Advancing Open-source World Modelscited by 144
  2. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  3. 2026Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Modelscited by 19
  4. 2026Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Modelscited by 15
  5. 2026VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Modelscited by 14
  6. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 12
  7. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  8. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  9. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  10. 2026Orca: The World is in Your Mindcited by 6
  11. 2026A Generalization Theory for JEPA-Based World Modelscited by 5
  12. 2026Multimodal Latent Reasoning via Predictive Embeddingscited by 4
38 more
  1. 2026The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHRcited by 3
  2. 2026Recursive Belief Vision Language Action Modelscited by 3
  3. 2026Self-Distillation of Hidden Layers for Self-Supervised Representation Learningcited by 3
  4. 2026Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstructioncited by 3
  5. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  6. 2026RISE: Adaptive Imagination for World Action Modelscited by 3
  7. 2026From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learningcited by 2
  8. 2026CGM-JEPA: Learning Consistent Continuous Glucose Monitor Representations via Predictive Self-Supervised Pretrainingcited by 2
  9. 2026AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modelingcited by 2
  10. 2026Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectoriescited by 2
  11. 2026CSI-JEPA: Towards Foundation Representations for Ubiquitous Sensing with Minimal Supervisioncited by 2
  12. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negativescited by 2
  13. 2026Unlocked Backpropagation using Wave Scatteringcited by 1
  14. 2026TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Modelscited by 1
  15. 2026PVI: Plug-in Visual Injection for Vision-Language-Action Modelscited by 1
  16. 2026Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMscited by 1
  17. 2026Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinementcited by 1
  18. 2026How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Predictioncited by 1
  19. 2026DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priorscited by 1
  20. 2026Latent Clarity: Bridging World-Model Kinematics to Semantic Manifolds for Video Anomaly Anticipationcited by 1
  21. 2026Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoningcited by 1
  22. 2026Beyond Representation Learning: A Systematic Study of Joint-Embedding Predictive Generation for 3D Brain MRIcited by 1
  23. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioningcited by 0
  24. 2026OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encodercited by 0
  25. 2026SARA: Semantically Adaptive Relational Alignment for Video Diffusion Modelscited by 0
  26. 2026HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learningcited by 0
  27. 2026Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?cited by 0
  28. 2026GRAG: Generic Response-Augmented Generation Framework for Personalized Conversational Systemscited by 0
  29. 2026LEViL: Label-Efficient Video Learning via Zero-Shot Distillation over VLM-Generated Pseudo-Label Spacescited by 0
  30. 2026VideoLatent: Video-Language Learning via Latent Self-Forcingcited by 0
  31. 2026P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecturecited by 0
  32. 2026VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretrainingcited by 0
  33. 2026Weight-Space Physics: Interpretable Hypernetworks for Lattice Quantum Field Theoriescited by 0
  34. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
  35. 2026CellWorld: From Gene-Level Reconstruction to Latent Cell Prediction in Spatial Transcriptomics Foundation Modelscited by 0
  36. 2026Toward a Causal Data Management Ecosystem for Decision Making and Agentic AIcited by 0
  37. 2026Energy-Structured Latent World Models with Neural Time Fields for Physically Constistent Open-World Motion Planningcited by 0
  38. 2026Scaling Representation Diversity: Modulated Attention and Reconstructive Regularization for Visual Groundingcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models
  2. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representations
  3. 2026ORION: ORthonormal Text Encoding for Universal VLM AdaptatION
  4. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
  5. 2026Unified Vision-Language Modeling via Concept Space Alignment
  6. 2022VL-BEiT: Generative Vision-Language Pretraining
  7. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
  8. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set