# A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures

arXiv: https://arxiv.org/abs/2602.03604 · HTML: https://mlsanity.com/paper/2602.03604/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 9 of its 55 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 9 of 9)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Intuitive physics understanding emerges from self-supervised pretraining on natural videos](https://mlsanity.com/paper/2502.11831/) (2025)
- [On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models](https://mlsanity.com/paper/1511.09249/) (2015)
- [What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?](https://mlsanity.com/paper/2512.24497/) (2025)
- [Diffusion Model Predictive Control](https://mlsanity.com/paper/2410.05364/) (2024)
- [Joint Embedding Predictive Architectures Focus on Slow Features](https://mlsanity.com/paper/2211.10831/) (2022)
- [stable-pretraining-v1: Foundation Model Research Made Simple](https://mlsanity.com/paper/2511.19484/) (2025)

## Built on by (most cited first, 14 of 14)

- [Hierarchical Planning with Latent World Models](https://mlsanity.com/paper/2604.03208/) (2026)
- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [Sensorimotor World Models: Perception for Action via Inverse Dynamics](https://mlsanity.com/paper/2606.20104/) (2026)
- [Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models](https://mlsanity.com/paper/2602.18639/) (2026)
- [LpWM: A Case for Sparse Representations in World Models](https://mlsanity.com/paper/2608.22764/) (2026)
- [ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulation](https://mlsanity.com/paper/2606.13877/) (2026)
- [ProWorld: Progress-Aware Hyperbolic World Models for Long-Horizon Visual Goal Reaching](https://mlsanity.com/paper/2608.01926/) (2026)
- [Is the reconstruction loss culprit? An attempt to outperform JEPA](https://mlsanity.com/paper/2603.14131/) (2026)
- [Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement](https://mlsanity.com/paper/2605.06298/) (2026)
- [PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment](https://mlsanity.com/paper/2605.17671/) (2026)
- [Data-driven Koopman mode approximation: A neural power iteration algorithm](https://mlsanity.com/paper/2608.26943/) (2026)
- [World Models for Cross-Machine CNC Transfer under Partial Sensor Overlap](https://mlsanity.com/paper/2609.16071/) (2026)
- [LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models](https://mlsanity.com/paper/2609.34375/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)

## Abstract (arXiv)

We present EB-JEPA, an open-source library for learning representations and world models using Joint-Embedding Predictive Architectures (JEPAs). JEPAs learn to predict in representation space rather than pixel space, avoiding the pitfalls of generative modeling while capturing semantically meaningful features suitable for downstream tasks. Our library provides modular, self-contained implementations that illustrate how representation learning techniques developed for image-level self-supervised learning can transfer to video, where temporal dynamics add complexity, and ultimately to action-conditioned world models, where the model must additionally learn to predict the effects of control inputs. Each example is designed for single-GPU training within a few hours, making energy-based self-supervised learning accessible for research and education. We provide ablations of JEA components on CIFAR-10. Probing these representations yields 91% accuracy, indicating that the model learns useful features. Extending to video, we include a multi-step prediction example on Moving MNIST that demonstrates how the same principles scale to temporal modeling. Finally, we show how these representations can drive action-conditioned world models, achieving a 97% planning success rate on the Two Rooms navigation task. Comprehensive ablations reveal the critical importance of each regularization component for preventing representation collapse. Code is available at https://github.com/facebookresearch/eb_jepa.