# Music-JEPA: Learning a World Model of Sound from Action

arXiv: https://arxiv.org/abs/2607.22000 · HTML: https://mlsanity.com/paper/2607.22000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)
- [A-JEPA: Joint-Embedding Predictive Architecture Can Listen](https://mlsanity.com/paper/2311.15830/) (2023)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [PAN: A World Model for General, Actionable, and Long-Horizon World Simulation](https://mlsanity.com/paper/2511.09057/) (2025)
- [Critique of World Model](https://mlsanity.com/paper/2507.05169/) (2025)
- [Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning](https://mlsanity.com/paper/2507.02915/) (2025)
- [Latent Geometry Beyond Search: Amortizing Planning in World Models](https://mlsanity.com/paper/2605.08732/) (2026)
- [Variance-Covariance Regularization Improves Representation Learning](https://mlsanity.com/paper/2306.13292/) (2023)
- [Closing the Train-Test Gap in World Models for Gradient-Based Planning](https://mlsanity.com/paper/2512.09929/) (2025)

## Built on by (most cited first, 2 of 2)

- [AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving](https://mlsanity.com/paper/2609.34085/) (2026)
- [Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models](https://mlsanity.com/paper/2610.07540/) (2026)

## Abstract (arXiv)

Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.