# LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors

arXiv: https://arxiv.org/abs/2205.15410 · HTML: https://mlsanity.com/paper/2205.15410/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 80 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 30 of 39)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Probabilistic Modeling for Human Mesh Recovery](https://mlsanity.com/paper/2108.11944/) (2021)
- [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022)
- [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Towards Accurate Markerless Human Shape and Pose Estimation over Time](https://mlsanity.com/paper/1707.07548/) (2017)
- [SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds](https://mlsanity.com/paper/2004.02774/) (2020)
- [EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera](https://mlsanity.com/paper/1908.11505/) (2019)
- [Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis](https://mlsanity.com/paper/2006.07364/) (2020)
- [MonoPerfCap: Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1708.02136/) (2017)
- [Neural Descent for Visual 3D Human Pose and Shape](https://mlsanity.com/paper/2008.06910/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception](https://mlsanity.com/paper/2109.05441/) (2021)
- [Harvesting Multiple Views for Marker-less 3D Human Pose Annotations](https://mlsanity.com/paper/1704.04793/) (2017)
- [A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation](https://mlsanity.com/paper/1602.03725/) (2016)

## Built on by (most cited first, 30 of 44)

- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment](https://mlsanity.com/paper/2404.00299/) (2024)
- [OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers](https://mlsanity.com/paper/2312.08985/) (2023)
- [I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions](https://mlsanity.com/paper/2312.08869/) (2023)
- [LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environment](https://mlsanity.com/paper/2403.13307/) (2024)
- [DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion](https://mlsanity.com/paper/2308.16682/) (2023)
- [Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors](https://mlsanity.com/paper/2312.02196/) (2023)
- [Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR](https://mlsanity.com/paper/2211.16951/) (2022)
- [LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment](https://mlsanity.com/paper/2402.17171/) (2024)
- [Human-centric Scene Understanding for 3D Large-scale Scenarios](https://mlsanity.com/paper/2307.14392/) (2023)
- [Gait Recognition in Large-scale Free Environment via Single LiDAR](https://mlsanity.com/paper/2211.12371/) (2022)
- [RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method](https://mlsanity.com/paper/2403.19501/) (2024)
- [Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels](https://mlsanity.com/paper/2603.02573/) (2026)
- [Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes](https://mlsanity.com/paper/2308.00628/) (2023)
- [PointHPS: Cascaded 3D Human Pose and Shape Estimation from Point Clouds](https://mlsanity.com/paper/2308.14492/) (2023)
- [Move-in-2D: 2D-Conditioned Human Motion Generation](https://mlsanity.com/paper/2412.13185/) (2024)
- [MOVIN: Real-time Motion Capture using a Single LiDAR](https://mlsanity.com/paper/2309.09314/) (2023)
- [LiveHPS++: Robust and Coherent Motion Capture in Dynamic Free Environment](https://mlsanity.com/paper/2407.09833/) (2024)
- [Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesis](https://mlsanity.com/paper/2412.02261/) (2024)
- [CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos](https://mlsanity.com/paper/2601.10632/) (2026)
- [HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR](https://mlsanity.com/paper/2409.04398/) (2024)
- [Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation](https://mlsanity.com/paper/2412.13454/) (2024)
- [UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Units](https://mlsanity.com/paper/2505.09393/) (2025)
- [Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokens](https://mlsanity.com/paper/2502.16175/) (2025)
- [A Unified Framework for Human-centric Point Cloud Video Understanding](https://mlsanity.com/paper/2403.20031/) (2024)
- [FreeCap: Hybrid Calibration-Free Motion Capture in Open Environments](https://mlsanity.com/paper/2411.04469/) (2024)
- [XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration](https://mlsanity.com/paper/2411.18377/) (2024)
- [HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes](https://mlsanity.com/paper/2403.02769/) (2024)
- [Towards Practical Human Motion Prediction with LiDAR Point Clouds](https://mlsanity.com/paper/2408.08202/) (2024)

## Abstract (arXiv)

We propose a multi-sensor fusion method for capturing challenging 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios, only using single LiDAR and 4 IMUs, which are set up conveniently and worn lightly. Specifically, to fully utilize the global geometry information captured by LiDAR and local dynamic motions captured by IMUs, we design a two-stage pose estimator in a coarse-to-fine manner, where point clouds provide the coarse body shape and IMU measurements optimize the local actions. Furthermore, considering the translation deviation caused by the view-dependent partial point cloud, we propose a pose-guided translation corrector. It predicts the offset between captured points and the real root locations, which makes the consecutive movements and trajectories more precise and natural. Moreover, we collect a LiDAR-IMU multi-modal mocap dataset, LIPD, with diverse human actions in long-range scenarios. Extensive quantitative and qualitative experiments on LIPD and other open datasets all demonstrate the capability of our approach for compelling motion capture in large-scale scenarios, which outperforms other methods by an obvious margin. We will release our code and captured dataset to stimulate future research.