# A Generalization Theory for JEPA-Based World Models

arXiv: https://arxiv.org/abs/2606.27014 · HTML: https://mlsanity.com/paper/2606.27014/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 19 of 19)

- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2204.00570/) (2022)
- [What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?](https://mlsanity.com/paper/2512.24497/) (2025)
- [How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks](https://mlsanity.com/paper/2407.03475/) (2024)
- [On the Generalization of Multi-modal Contrastive Learning](https://mlsanity.com/paper/2306.04272/) (2023)
- [When Does LeJEPA Learn a World Model?](https://mlsanity.com/paper/2605.26379/) (2026)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [Value-guided action planning with JEPA world models](https://mlsanity.com/paper/2601.00844/) (2025)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density](https://mlsanity.com/paper/2510.05949/) (2025)
- [Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining](https://mlsanity.com/paper/2407.00935/) (2024)
- [FF-JEPA: Long-Horizon Planning in World Models with Latent Planners](https://mlsanity.com/paper/2606.09311/) (2026)
- [Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.01317/) (2025)

## Built on by (most cited first, 5 of 5)

- [QQWorld: Quantile-Quantile Matching for World Model Regularization](https://mlsanity.com/paper/2607.28415/) (2026)
- [What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations](https://mlsanity.com/paper/2607.27017/) (2026)
- [Finite-Sample Metric Non-Collapse for Geometrically Supervised Latent World Models in Control](https://mlsanity.com/paper/2608.07265/) (2026)
- [LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models](https://mlsanity.com/paper/2609.34375/) (2026)
- [Beyond a single latent space: a dual-latent world model for long-horizon planning](https://mlsanity.com/paper/2609.37644/) (2026)

## Abstract (arXiv)

Joint Embedding Predictive Architectures (JEPAs) have emerged as a promising paradigm for world modeling by predicting future representations rather than reconstructing observations. Despite their empirical success, the theoretical understanding of JEPA-based world models remains limited. In this paper, we develop the first generalization theory for JEPA-based world models. We formulate JEPA pretraining as a conditional spectral graph learning problem and show that the JEPA objective is equivalent to a low-rank factorization of an action-conditioned co-occurrence matrix. Building on this characterization, we establish a connection between JEPA pretraining error and downstream planning regret, leading to a finite-sample generalization bound for JEPA-based world models. Our analysis reveals an approximation-sample error trade-off governed by latent dimensionality, providing insights into latent- versus input-level prediction. The theoretical trade-off motivates us to investigate a simple Multi-Depth JEPA extension that combines representations from different network depths. In validation experiments, latent-level prediction exhibits greater robustness under unpredictable observations and difficult planning tasks. Multi-depth representations further improve planning success over single-depth representations and a dimensionality-matched control.