# Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users

arXiv: https://arxiv.org/abs/2209.05753 · HTML: https://mlsanity.com/paper/2209.05753/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2022.

## Built on by (most cited first, 21 of 21)

- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model](https://mlsanity.com/paper/2304.08577/) (2023)
- [TLControl: Trajectory and Language Control for Human Motion Synthesis](https://mlsanity.com/paper/2311.17135/) (2023)
- [EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation](https://mlsanity.com/paper/2312.02256/) (2023)
- [QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors](https://mlsanity.com/paper/2306.05666/) (2023)
- [Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture](https://mlsanity.com/paper/2309.00310/) (2023)
- [A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals](https://mlsanity.com/paper/2404.04890/) (2024)
- [DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers](https://mlsanity.com/paper/2402.09211/) (2024)
- [Pose-aware Attention Network for Flexible Motion Retargeting by Body Part](https://mlsanity.com/paper/2306.08006/) (2023)
- [Physics-based Motion Retargeting from Sparse Inputs](https://mlsanity.com/paper/2307.01938/) (2023)
- [Diffusion-based Human Motion Style Transfer with Semantic Guidance](https://mlsanity.com/paper/2405.06646/) (2024)
- [HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA](https://mlsanity.com/paper/2511.01463/) (2025)
- [Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions](https://mlsanity.com/paper/2306.04161/) (2023)
- [DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization](https://mlsanity.com/paper/2406.14567/) (2024)
- [SMGDiff: Soccer Motion Generation using diffusion probabilistic models](https://mlsanity.com/paper/2411.16216/) (2024)
- [XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration](https://mlsanity.com/paper/2411.18377/) (2024)
- [Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals](https://mlsanity.com/paper/2308.15839/) (2023)
- [DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera](https://mlsanity.com/paper/2508.06139/) (2025)
- [Robo-Saber: Generating and Simulating Virtual Reality Players](https://mlsanity.com/paper/2602.18319/) (2026)
- [Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robot](https://mlsanity.com/paper/2604.21351/) (2026)
- [UniMo: Unifying Human and Animal Motion Generation](https://mlsanity.com/paper/2609.12342/) (2026)

## Abstract (arXiv)

Animating an avatar that reflects a user's action in the VR world enables natural interactions with the virtual environment. It has the potential to allow remote users to communicate and collaborate in a way as if they met in person. However, a typical VR system provides only a very sparse set of up to three positional sensors, including a head-mounted display (HMD) and optionally two hand-held controllers, making the estimation of the user's full-body movement a difficult problem. In this work, we present a data-driven physics-based method for predicting the realistic full-body movement of the user according to the transformations of these VR trackers and simulating an avatar character to mimic such user actions in the virtual world in real-time. We train our system using reinforcement learning with carefully designed pretraining processes to ensure the success of the training and the quality of the simulation. We demonstrate the effectiveness of the method with an extensive set of examples.