# Learning Robot Manipulation from Audio World Models

arXiv: https://arxiv.org/abs/2512.08405 · HTML: https://mlsanity.com/paper/2512.08405/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 9 Dec 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 9 of 9)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [FLARE: Robot Learning with Implicit World Modeling](https://mlsanity.com/paper/2505.15659/) (2025)
- [Video Generators are Robot Policies](https://mlsanity.com/paper/2508.00795/) (2025)
- [Affordance-based Robot Manipulation with Flow Matching](https://mlsanity.com/paper/2409.01083/) (2024)
- [FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait](https://mlsanity.com/paper/2412.01064/) (2024)
- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.04150/) (2023)

## Built on by (most cited first, 3 of 3)

- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)

## Abstract (arXiv)

World models have demonstrated impressive performance on robotic learning tasks. Many such tasks inherently demand multimodal reasoning; for example, filling a bottle with water will lead to visual information alone being ambiguous or incomplete, thereby requiring reasoning over the temporal evolution of audio, accounting for its underlying physical properties and pitch patterns. In this paper, we propose a generative latent flow matching model to anticipate future audio observations, enabling the system to reason about long-term consequences when integrated into a robot policy. We demonstrate the superior capabilities of our system through two manipulation tasks that require perceiving in-the-wild audio or music signals, compared to methods without future lookahead. We further emphasize that successful robot action learning for these tasks relies not merely on multi-modal input, but critically on the accurate prediction of future audio states that embody intrinsic rhythmic patterns.