mlsanityData to 7 Oct 2026

Revisiting Feature Prediction for Learning Visual Representations from Video

arXiv 2404.08471 ↗First posted 15 Feb 2024cs.CV cs.AI cs.LGcited by 490

On the map: World models → Latent predictive world models

Abstract (arXiv)

This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model's parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.

Key facts. Cited by 490 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 63 of its 83 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

63 of 83 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan

Other references · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  13. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  14. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  15. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  16. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  18. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  19. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  20. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  21. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  22. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  23. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  24. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  25. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  26. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  27. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  28. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  29. 2016Learning Representations for Automatic Colorizationcited by 251
  30. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  31. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  32. 2021Anticipative Video Transformercited by 197
  33. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  34. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  35. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  36. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  37. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  38. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 104; 2026: 339.

490 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  5. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  6. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  7. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  8. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 76
  9. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  10. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  11. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  12. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
38 more
  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  2. 20253D and 4D World Modeling: A Surveycited by 64
  3. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  4. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  5. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  6. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 49
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  8. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
  9. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  10. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  11. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 40
  12. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 38
  13. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  14. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  15. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  16. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  17. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 35
  18. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  19. 2026Temporal Straightening for Latent Planningcited by 32
  20. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  21. 2024Scaling 4D Representationscited by 30
  22. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  23. 2026When Does LeJEPA Learn a World Model?cited by 27
  24. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 26
  25. 2025Scaling up self-supervised learning for improved surgical foundation modelscited by 25
  26. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 25
  27. 2025Critique of World Modelcited by 24
  28. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  29. 2025Embodied AI: From LLMs to World Modelscited by 23
  30. 2026JEPA-VLA: Video Predictive Embedding is Needed for VLA Modelscited by 22
  31. 2024Beyond FVD: Enhanced Evaluation Metrics for Video Generation Qualitycited by 21
  32. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Videocited by 20
  33. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 20
  34. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Modelscited by 20
  35. 2024Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learningcited by 19
  36. 2025Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networkscited by 19
  37. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
  38. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19

50 most cited of 490 All 490 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives
  2. 2024Learning and Leveraging World Models in Visual Representation Learning
  3. 2026Latent Video Prediction Learns Better World Models
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  5. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  6. 2023Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers
  7. 2024Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learning
  8. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model