# Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model

arXiv: https://arxiv.org/abs/2304.08577 · HTML: https://mlsanity.com/paper/2304.08577/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022)

## Built on by (most cited first, 30 of 107)

- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild](https://mlsanity.com/paper/2406.09905/) (2024)
- [Optimizing Diffusion Noise Can Serve As Universal Motion Priors](https://mlsanity.com/paper/2312.11994/) (2023)
- [Inter-X: Towards Versatile Human-Human Interaction Analysis](https://mlsanity.com/paper/2312.16051/) (2023)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [ReGenNet: Towards Human Action-Reaction Synthesis](https://mlsanity.com/paper/2403.11882/) (2024)
- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis](https://mlsanity.com/paper/2304.11118/) (2023)
- [LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation](https://mlsanity.com/paper/2309.09294/) (2023)
- [MaskControl: Spatio-Temporal Control for Masked Motion Synthesis](https://mlsanity.com/paper/2410.10780/) (2024)
- [EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere](https://mlsanity.com/paper/2308.06493/) (2023)
- [Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style](https://mlsanity.com/paper/2403.06365/) (2024)
- [DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion](https://mlsanity.com/paper/2308.16682/) (2023)
- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [EgoLM: Multi-Modal Language Model of Egocentric Motions](https://mlsanity.com/paper/2409.18127/) (2024)
- [HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device](https://mlsanity.com/paper/2409.13426/) (2024)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [Stratified Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2405.20786/) (2024)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis](https://mlsanity.com/paper/2508.02106/) (2025)
- [Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging](https://mlsanity.com/paper/2404.19541/) (2024)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input](https://mlsanity.com/paper/2504.08449/) (2025)
- [A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals](https://mlsanity.com/paper/2404.04890/) (2024)
- [Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots](https://mlsanity.com/paper/2408.07295/) (2024)
- [HERO: Human Reaction Generation from Videos](https://mlsanity.com/paper/2503.08270/) (2025)

## Abstract (arXiv)

With the recent surge in popularity of AR/VR applications, realistic and accurate control of 3D full-body avatars has become a highly demanded feature. A particular challenge is that only a sparse tracking signal is available from standalone HMDs (Head Mounted Devices), often limited to tracking the user's head and wrists. While this signal is resourceful for reconstructing the upper body motion, the lower body is not tracked and must be synthesized from the limited information provided by the upper body joints. In this paper, we present AGRoL, a novel conditional diffusion model specifically designed to track full bodies given sparse upper-body tracking signals. Our model is based on a simple multi-layer perceptron (MLP) architecture and a novel conditioning scheme for motion data. It can predict accurate and smooth full-body motion, particularly the challenging lower body movement. Unlike common diffusion architectures, our compact architecture can run in real-time, making it suitable for online body-tracking applications. We train and evaluate our model on AMASS motion capture dataset, and demonstrate that our approach outperforms state-of-the-art methods in generated motion accuracy and smoothness. We further justify our design choices through extensive experiments and ablation studies.