# Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression

arXiv: https://arxiv.org/abs/2502.04296 · HTML: https://mlsanity.com/paper/2502.04296/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2025.

## Built on by (most cited first, 23 of 23)

- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [RoboScape: Physics-informed Embodied World Model](https://mlsanity.com/paper/2506.23135/) (2025)
- [BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks](https://mlsanity.com/paper/2602.03793/) (2026)
- [PlayWorld: Learning Robot World Models from Autonomous Play](https://mlsanity.com/paper/2603.09030/) (2026)
- [ORV: 4D Occupancy-centric Robot Video Generation](https://mlsanity.com/paper/2506.03079/) (2025)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [OAT: Ordered Action Tokenization](https://mlsanity.com/paper/2602.04215/) (2026)
- [H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos](https://mlsanity.com/paper/2512.09406/) (2025)
- [Robotic VLA Benefits from Joint Learning with Motion Image Diffusion](https://mlsanity.com/paper/2512.18007/) (2025)
- [Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos](https://mlsanity.com/paper/2511.12882/) (2025)
- [Flexible Multitask Learning with Factorized Diffusion Policy](https://mlsanity.com/paper/2512.21898/) (2025)
- [DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation](https://mlsanity.com/paper/2608.13489/) (2026)
- [RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation](https://mlsanity.com/paper/2607.01060/) (2026)
- [BWM: A Low-Cost High-Fidelity World Simulator for Robot Learning](https://mlsanity.com/paper/2607.29302/) (2026)
- [Localized Graph-Based Neural Dynamics Models for Terrain Manipulation](https://mlsanity.com/paper/2503.23270/) (2025)
- [Learning Robust Visuomotor Policies via Dynamics Alignment](https://mlsanity.com/paper/2510.27114/) (2025)
- [From Generation to Simulation: How Far Are World Models from Being True Simulators?](https://mlsanity.com/paper/2608.23070/) (2026)
- [ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies](https://mlsanity.com/paper/2610.04607/) (2026)
- [SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models](https://mlsanity.com/paper/2610.15645/) (2026)

## Abstract (arXiv)

We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics to generate high-quality data and evaluation in scaling robot learning. Building interactive video world models and policies for robotics is difficult due to the challenge of handling diverse settings while maintaining computational efficiency to run in real time. HMA uses heterogeneous pre-training from observations and action sequences across different robotic embodiments, domains, and tasks. HMA uses masked autoregression to generate quantized or soft tokens for video predictions. ourshort achieves better visual fidelity and controllability than the previous robotic video generation models with 15 times faster speed in the real world. After post-training, this model can be used as a video simulator from low-level action inputs for evaluating policies and generating synthetic data. See this link https://liruiw.github.io/hma for more information.