# Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices

arXiv: https://arxiv.org/abs/2209.11478 · HTML: https://mlsanity.com/paper/2209.11478/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2022.

## Built on by (most cited first, 13 of 13)

- [QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors](https://mlsanity.com/paper/2306.05666/) (2023)
- [SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data](https://mlsanity.com/paper/2311.02191/) (2023)
- [HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations](https://mlsanity.com/paper/2308.11261/) (2023)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [SimpleEgo: Predicting Probabilistic Body Pose from Egocentric Cameras](https://mlsanity.com/paper/2401.14785/) (2024)
- [DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers](https://mlsanity.com/paper/2402.09211/) (2024)
- [Self-Avatar Animation in Virtual Reality: Impact of Motion Signals Artifacts on the Full-Body Pose Reconstruction](https://mlsanity.com/paper/2404.18628/) (2024)
- [DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization](https://mlsanity.com/paper/2406.14567/) (2024)
- [Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals](https://mlsanity.com/paper/2308.15839/) (2023)
- [Environment-aware Motion Matching](https://mlsanity.com/paper/2510.22632/) (2025)
- [Step2Motion: Locomotion Reconstruction from Pressure Sensing Insoles](https://mlsanity.com/paper/2510.22712/) (2025)
- [Dancing Points: Synthesizing Ballroom Dancing with Three-Point Inputs](https://mlsanity.com/paper/2601.02096/) (2026)
- [STyMo: Fast and Controllable Few-Shot Motion Style Transfer](https://mlsanity.com/paper/2609.04500/) (2026)

## Abstract (arXiv)

The animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self-avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer-grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstruction the user pose from such sparse data is ill-defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time-warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self-avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.