mlsanityData to 14 Oct 2026

Learning and Leveraging World Models in Visual Representation Learning

arXiv 2403.00504 ↗First posted 1 Mar 2024cs.CV cs.AI cs.LGcited by 72

On the map: World models → Latent predictive world models

Abstract (arXiv)

Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising self-supervised approach that learns by leveraging a world model. While previously limited to predicting missing parts of an input, we explore how to generalize the JEPA prediction task to a broader set of corruptions. We introduce Image World Models, an approach that goes beyond masked image modeling and learns to predict the effect of global photometric transformations in latent space. We study the recipe of learning performant IWMs and show that it relies on three key aspects: conditioning, prediction difficulty, and capacity. Additionally, we show that the predictive world model learned by IWM can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM world model matches or surpasses the performance of previous self-supervised methods. Finally, we show that learning with an IWM allows one to control the abstraction level of the learned representations, learning invariant representations such as contrastive methods, or equivariant representations such as masked image modelling.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

42 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan
  2. 2022Learning Symmetric Embeddings for Equivariant World Models24 Apr
  3. 2019Contrastive Learning of Structured World Models27 Nov

Other references · 39

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
27 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2017Large Batch Training of Convolutional Networkscited by 535
  12. 2023Learning Interactive Real-World Simulatorscited by 458
  13. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  14. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  15. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  16. 2020Whitening for Self-Supervised Representation Learningcited by 279
  17. 2021Decoupled Contrastive Learningcited by 143
  18. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  19. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  20. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  21. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  22. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 91
  23. 2021Equivariant Contrastive Learningcited by 66
  24. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
  25. 2022Neural Manifold Clustering and Embeddingcited by 44
  26. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 44
  27. 2023Amortised Invariance Learning for Contrastive Self-Supervisioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 29; 2026: 25.

72 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2025Galileo: Learning Global & Local Features of Many Remote Sensing Modalitiescited by 43
  3. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  4. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 20
  5. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 20
  6. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learningcited by 19
  7. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
  8. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  9. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  10. 2025EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidancecited by 15
  11. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  12. 2024ACT-Bench: Towards Action Controllable World Models for Autonomous Drivingcited by 14
38 more
  1. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  2. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  3. 2024On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theorycited by 10
  4. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  5. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  6. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 9
  7. 2024Self-supervised contrastive learning performs non-linear system identificationcited by 9
  8. 2025ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learningcited by 8
  9. 2025Flopping for FLOPs: Leveraging equivariance for computational efficiencycited by 8
  10. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  11. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  12. 2025EgoAgent: A Joint Predictive Agent Model in Egocentric Worldscited by 7
  13. 2025seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Modelscited by 7
  14. 2024Affine steerers for structured keypoint descriptioncited by 6
  15. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 6
  16. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  17. 2026GeoWorld: Geometric World Modelscited by 6
  18. 2024In-Context Symmetries: Self-Supervised Learning through Contextual World Modelscited by 5
  19. 2025Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformerscited by 4
  20. 2025Learning Actionable World Models for Industrial Process Controlcited by 4
  21. 2025Super Encoding Network: Recursive Association of Multi-Modal Encoders for Video Understandingcited by 4
  22. 2026AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modelingcited by 4
  23. 2026Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translationcited by 4
  24. 2025Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learningcited by 3
  25. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  26. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  27. 2024Object-centric proto-symbolic behavioural reasoning from pixelscited by 2
  28. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  29. 2025Adaptive Bayesian Single-Shot Quantum Sensingcited by 2
  30. 2025Learning State-Space Models of Dynamic Systems from Arbitrary Data using Joint Embedding Predictive Architecturescited by 2
  31. 2025Diffusion-Based Action Recognition Generalizes to Untrained Domainscited by 2
  32. 2025Midway Network: Learning Representations for Recognition and Motion from Latent Dynamicscited by 2
  33. 2025X-WIN: Building Chest Radiograph World Model via Predictive Sensingcited by 2
  34. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Predictioncited by 2
  35. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representationscited by 2
  36. 2025A Data-Driven Discretized CS:GO Simulation Environment to Facilitate Strategic Multi-Agent Planning Researchcited by 1
  37. 2025Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platformcited by 1
  38. 2026Self-Supervised Learning from Structural Invariancecited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
  2. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
  3. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
  4. 2026V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
  5. 2022Joint Embedding Predictive Architectures Focus on Slow Features
  6. 2026No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models
  7. 2026Latent Video Prediction Learns Better World Models
  8. 2026Frozen in a Frame: The Velocity Blind Spot in JEPA World Models