# From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models

arXiv: https://arxiv.org/abs/2504.05265 · HTML: https://mlsanity.com/paper/2504.05265/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 66 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022)
- [AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing](https://mlsanity.com/paper/2207.13784/) (2022)
- [Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model](https://mlsanity.com/paper/2304.08577/) (2023)
- [Rolling Diffusion Models](https://mlsanity.com/paper/2402.09470/) (2024)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [Seamless Human Motion Composition with Blended Positional Encodings](https://mlsanity.com/paper/2402.15509/) (2024)
- [Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation](https://mlsanity.com/paper/2203.15720/) (2022)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Estimating Body and Hand Motion in an Ego-sensed World](https://mlsanity.com/paper/2410.03665/) (2024)
- [IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds](https://mlsanity.com/paper/2304.12518/) (2023)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesis](https://mlsanity.com/paper/2307.15042/) (2023)
- [BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis](https://mlsanity.com/paper/2304.11118/) (2023)
- [Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling](https://mlsanity.com/paper/2308.08855/) (2023)
- [Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors](https://mlsanity.com/paper/2302.04860/) (2023)
- [EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere](https://mlsanity.com/paper/2308.06493/) (2023)

## Built on by (most cited first, 11 of 11)

- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR](https://mlsanity.com/paper/2603.04090/) (2026)
- [KineST: A Kinematics-guided Spatiotemporal State Space Model for Human Motion Tracking from Sparse Signals](https://mlsanity.com/paper/2512.16791/) (2025)
- [EgoExoMoCap: Distributed Ego-Exo Human Motion Capture](https://mlsanity.com/paper/2607.15868/) (2026)
- [OmniEgoCap: Camera-Agnostic Sequence-Level Egocentric Motion Reconstruction](https://mlsanity.com/paper/2512.19283/) (2025)
- [EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcing](https://mlsanity.com/paper/2605.13041/) (2026)
- [AtomicMotion: Learning Human Motion From Different Human Parts](https://mlsanity.com/paper/2605.22631/) (2026)
- [MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations](https://mlsanity.com/paper/2606.23000/) (2026)
- [IMAGIN-4D: Image-Guided Controllable Interaction Generation](https://mlsanity.com/paper/2606.23675/) (2026)
- [EgoGVAE: Ego-body Mesh Reconstruction via Guided Variational Autoencoder](https://mlsanity.com/paper/2607.27755/) (2026)

## Abstract (arXiv)

In extended reality (XR), generating full-body motion of the users is important to understand their actions, drive their virtual avatars for social interaction, and convey a realistic sense of presence. While prior works focused on spatially sparse and always-on input signals from motion controllers, many XR applications opt for vision-based hand tracking for reduced user friction and better immersion. Compared to controllers, hand tracking signals are less accurate and can even be missing for an extended period of time. To handle such unreliable inputs, we present Rolling Prediction Model (RPM), an online and real-time approach that generates smooth full-body motion from temporally and spatially sparse input signals. Our model generates 1) accurate motion that matches the inputs (i.e., tracking mode) and 2) plausible motion when inputs are missing (i.e., synthesis mode). More importantly, RPM generates seamless transitions from tracking to synthesis, and vice versa. To demonstrate the practical importance of handling noisy and missing inputs, we present GORP, the first dataset of realistic sparse inputs from a commercial virtual reality (VR) headset with paired high quality body motion ground truth. GORP provides >14 hours of VR gameplay data from 28 people using motion controllers (spatially sparse) and hand tracking (spatially and temporally sparse). We benchmark RPM against the state of the art on both synthetic data and GORP to highlight how we can bridge the gap for real-world applications with a realistic dataset and by handling unreliable input signals. Our code, pretrained models, and GORP dataset are available in the project webpage.