# Learning Predictive Visuomotor Coordination

arXiv: https://arxiv.org/abs/2503.23300 · HTML: https://mlsanity.com/paper/2503.23300/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 30 Mar 2025.

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild](https://mlsanity.com/paper/2406.09905/) (2024)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [HumanMAC: Masked Motion Completion for Human Motion Prediction](https://mlsanity.com/paper/2302.03665/) (2023)
- [OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation](https://mlsanity.com/paper/2410.11792/) (2024)
- [ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation](https://mlsanity.com/paper/2405.02292/) (2024)
- [Learning Generalizable Manipulation Policies with Object-Centric 3D Representations](https://mlsanity.com/paper/2310.14386/) (2023)
- [Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning](https://mlsanity.com/paper/2501.06994/) (2025)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids](https://mlsanity.com/paper/2502.20396/) (2025)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)

## Built on by (most cited first, 2 of 2)

- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [Learning Humanoid Navigation from Human Data](https://mlsanity.com/paper/2604.00416/) (2026)

## Abstract (arXiv)

Understanding and predicting human visuomotor coordination is crucial for applications in robotics, human-computer interaction, and assistive technologies. This work introduces a forecasting-based task for visuomotor modeling, where the goal is to predict head pose, gaze, and upper-body motion from egocentric visual and kinematic observations. We propose a textitVisuomotor Coordination Representation (VCR) that learns structured temporal dependencies across these multimodal signals. We extend a diffusion-based motion modeling framework that integrates egocentric vision and kinematic sequences, enabling temporally coherent and accurate visuomotor predictions. Our approach is evaluated on the large-scale EgoExo4D dataset, demonstrating strong generalization across diverse real-world activities. Our results highlight the importance of multimodal integration in understanding visuomotor coordination, contributing to research in visuomotor learning and human behavior modeling. Project Page: https://vjwq.github.io/VCR/.