# HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR

arXiv: https://arxiv.org/abs/2203.09215 · HTML: https://mlsanity.com/paper/2203.09215/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 16 of 16)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017)
- [AtLoc: Attention Guided Camera Localization](https://mlsanity.com/paper/1909.03557/) (2019)
- [VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry](https://mlsanity.com/paper/1804.08366/) (2018)
- [LO-Net: Deep Real-time Lidar Odometry](https://mlsanity.com/paper/1904.08242/) (2019)
- [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021)
- [F-LOAM: Fast LiDAR Odometry And Mapping](https://mlsanity.com/paper/2107.00822/) (2021)
- [PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections](https://mlsanity.com/paper/1809.03605/) (2018)
- [Loam_livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV](https://mlsanity.com/paper/1909.06700/) (2019)
- [FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras](https://mlsanity.com/paper/1610.09534/) (2016)
- [LIC-Fusion: LiDAR-Inertial-Camera Odometry](https://mlsanity.com/paper/1909.04102/) (2019)
- [LocNet: Global localization in 3D point clouds for mobile vehicles](https://mlsanity.com/paper/1712.02165/) (2017)
- [Robust Odometry and Mapping for Multi-LiDAR Systems with Online Extrinsic Calibration](https://mlsanity.com/paper/2010.14294/) (2020)

## Built on by (most cited first, 30 of 38)

- [EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild](https://mlsanity.com/paper/2308.16894/) (2023)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023)
- [SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments](https://mlsanity.com/paper/2303.09095/) (2023)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement](https://mlsanity.com/paper/2406.19353/) (2024)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environment](https://mlsanity.com/paper/2403.13307/) (2024)
- [Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors](https://mlsanity.com/paper/2312.02196/) (2023)
- [Joint Optimization for 4D Human-Scene Reconstruction in the Wild](https://mlsanity.com/paper/2501.02158/) (2025)
- [Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion](https://mlsanity.com/paper/2205.02830/) (2022)
- [CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions](https://mlsanity.com/paper/2303.17948/) (2023)
- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)
- [LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment](https://mlsanity.com/paper/2402.17171/) (2024)
- [Human-centric Scene Understanding for 3D Large-scale Scenarios](https://mlsanity.com/paper/2307.14392/) (2023)
- [MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors](https://mlsanity.com/paper/2403.17610/) (2024)
- [Synergistic Global-space Camera and Human Reconstruction from Videos](https://mlsanity.com/paper/2405.14855/) (2024)
- [Gait Recognition in Large-scale Free Environment via Single LiDAR](https://mlsanity.com/paper/2211.12371/) (2022)
- [RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method](https://mlsanity.com/paper/2403.19501/) (2024)
- [RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark](https://mlsanity.com/paper/2407.13930/) (2024)
- [PointHPS: Cascaded 3D Human Pose and Shape Estimation from Point Clouds](https://mlsanity.com/paper/2308.14492/) (2023)
- [EgoHDM: An Online Egocentric-Inertial Human Motion Capture, Localization, and Dense Mapping System](https://mlsanity.com/paper/2409.00343/) (2024)
- [ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos](https://mlsanity.com/paper/2504.13167/) (2025)
- [Ego3DT: Tracking Every 3D Object in Ego-centric Videos](https://mlsanity.com/paper/2410.08530/) (2024)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR](https://mlsanity.com/paper/2409.04398/) (2024)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition](https://mlsanity.com/paper/2403.15444/) (2024)
- [NeuroGauss4D-PCI: 4D Neural Fields and Gaussian Deformation Fields for Point Cloud Interpolation](https://mlsanity.com/paper/2405.14241/) (2024)
- [HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes](https://mlsanity.com/paper/2403.02769/) (2024)

## Abstract (arXiv)

We propose Human-centered 4D Scene Capture (HSC4D) to accurately and efficiently create a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, and rich interactions between humans and environments. Using only body-mounted IMUs and LiDAR, HSC4D is space-free without any external devices' constraints and map-free without pre-built maps. Considering that IMUs can capture human poses but always drift for long-period use, while LiDAR is stable for global localization but rough for local positions and orientations, HSC4D makes both sensors complement each other by a joint optimization and achieves promising results for long-term capture. Relationships between humans and environments are also explored to make their interaction more realistic. To facilitate many down-stream tasks, like AR, VR, robots, autonomous driving, etc., we propose a dataset containing three large scenes (1k-5k m2) with accurate dynamic human motions and locations. Diverse scenarios (climbing gym, multi-story building, slope, etc.) and challenging human activities (exercising, walking up/down stairs, climbing, etc.) demonstrate the effectiveness and the generalization ability of HSC4D. The dataset and code are available at http://www.lidarhumanmotion.net/hsc4d/.