# SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

arXiv: https://arxiv.org/abs/2311.02191 · HTML: https://mlsanity.com/paper/2311.02191/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing](https://mlsanity.com/paper/2207.13784/) (2022)
- [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017)
- [QuaterNet: A Quaternion-based Recurrent Model for Human Motion](https://mlsanity.com/paper/1805.06485/) (2018)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [Unpaired Motion Style Transfer from Video to Animation](https://mlsanity.com/paper/2005.05751/) (2020)
- [QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars](https://mlsanity.com/paper/2209.09391/) (2022)
- [Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation](https://mlsanity.com/paper/2203.15720/) (2022)
- [FLAG: Flow-based 3D Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2203.05789/) (2022)
- [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)
- [LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals](https://mlsanity.com/paper/2103.01500/) (2021)
- [SOMA: Solving Optical Marker-Based MoCap Automatically](https://mlsanity.com/paper/2110.04431/) (2021)
- [HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media](https://mlsanity.com/paper/2110.07235/) (2021)
- [Learning Local Shape Descriptors from Part Correspondences With Multi-view Convolutional Networks](https://mlsanity.com/paper/1706.04496/) (2017)
- [IKFlow: Generating Diverse Inverse Kinematics Solutions](https://mlsanity.com/paper/2111.08933/) (2021)
- [Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices](https://mlsanity.com/paper/2209.11478/) (2022)
- [Pose Representations for Deep Skeletal Animation](https://mlsanity.com/paper/2111.13907/) (2021)
- [Neural Inverse Kinematics](https://mlsanity.com/paper/2205.10837/) (2022)
- [Learning-based pose edition for efficient and interactive design](https://mlsanity.com/paper/2107.00397/) (2021)
- [AvatarGo: Plug and Play self-avatars for VR](https://mlsanity.com/paper/2209.11482/) (2022)
- [Animation Fidelity in Self-Avatars: Impact on User Performance and Sense of Agency](https://mlsanity.com/paper/2304.05334/) (2023)

## Built on by (most cited first, 21 of 21)

- [GENMO: A GENeralist Model for Human MOtion](https://mlsanity.com/paper/2505.01425/) (2025)
- [I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions](https://mlsanity.com/paper/2312.08869/) (2023)
- [EgoLM: Multi-Modal Language Model of Egocentric Motions](https://mlsanity.com/paper/2409.18127/) (2024)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [MobilePoser: Real-Time Full-Body Pose Estimation and 3D Human Translation from IMUs in Mobile Consumer Devices](https://mlsanity.com/paper/2504.12492/) (2025)
- [Deep learning for 3D human pose estimation and mesh recovery: A survey](https://mlsanity.com/paper/2402.18844/) (2024)
- [EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling](https://mlsanity.com/paper/2412.10235/) (2024)
- [UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Units](https://mlsanity.com/paper/2505.09393/) (2025)
- [DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization](https://mlsanity.com/paper/2406.14567/) (2024)
- [ASMR: Adaptive Skeleton-Mesh Rigging and Skinning via 2D Generative Prior](https://mlsanity.com/paper/2503.13579/) (2025)
- [Physical Self-Supervised Learning: IMU Sensing without Manual Labels](https://mlsanity.com/paper/2607.18361/) (2026)
- [ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling](https://mlsanity.com/paper/2410.06963/) (2024)
- [Shape-aware Inertial Poser: Motion Tracking for Humans with Diverse Shapes Using Sparse Inertial Sensors](https://mlsanity.com/paper/2510.17101/) (2025)
- [Step2Motion: Locomotion Reconstruction from Pressure Sensing Insoles](https://mlsanity.com/paper/2510.22712/) (2025)
- [Garment Inertial Denoiser (GID): Endowing Accurate Motion Capture via Loose IMU Denoiser](https://mlsanity.com/paper/2601.01360/) (2026)
- [Dancing Points: Synthesizing Ballroom Dancing with Three-Point Inputs](https://mlsanity.com/paper/2601.02096/) (2026)
- [EgoPoseVR: Spatiotemporal Multi-Modal Reasoning for Egocentric Full-Body Pose in Virtual Reality](https://mlsanity.com/paper/2602.05590/) (2026)
- [Int3DNet: Scene-Motion Cross Attention Network for 3D Intention Prediction in Mixed Reality](https://mlsanity.com/paper/2603.13355/) (2026)
- [Amortized Inverse Kinematics via Graph Attention for Real-Time Human Avatar Animation](https://mlsanity.com/paper/2604.16629/) (2026)
- [Classifying daily activities needs posture, reconstructing them needs motion](https://mlsanity.com/paper/2607.13216/) (2026)
- [EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ](https://mlsanity.com/paper/2610.18872/) (2026)

## Abstract (arXiv)

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions, though, it is important to minimize the number of sensors attached to the subject's body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.