# HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR arXiv: https://arxiv.org/abs/2409.04398 ยท HTML: https://mlsanity.com/paper/2409.04398/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 84 references that are in the map. - First posted on arXiv 6 Sep 2024. ## Builds on (most cited first, 30 of 37) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019) - [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019) - [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018) - [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020) - [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021) - [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022) - [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021) - [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021) - [Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction](https://mlsanity.com/paper/2007.11432/) (2020) - [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017) - [Towards Accurate Markerless Human Shape and Pose Estimation over Time](https://mlsanity.com/paper/1707.07548/) (2017) - [AtLoc: Attention Guided Camera Localization](https://mlsanity.com/paper/1909.03557/) (2019) - [You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions](https://mlsanity.com/paper/1904.09882/) (2019) - [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019) - [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018) - [VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry](https://mlsanity.com/paper/1804.08366/) (2018) - [LO-Net: Deep Real-time Lidar Odometry](https://mlsanity.com/paper/1904.08242/) (2019) - [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022) - [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023) - [SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments](https://mlsanity.com/paper/2303.09095/) (2023) - [Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video](https://mlsanity.com/paper/1603.07763/) (2016) - [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021) - [F-LOAM: Fast LiDAR Odometry And Mapping](https://mlsanity.com/paper/2107.00822/) (2021) - [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022) - [4D Human Body Capture from Egocentric Video via 3D Scene Grounding](https://mlsanity.com/paper/2011.13341/) (2020) - [HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR](https://mlsanity.com/paper/2203.09215/) (2022) - [LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds](https://mlsanity.com/paper/2203.14698/) (2022) - [PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections](https://mlsanity.com/paper/1809.03605/) (2018) ## Built on by (most cited first, 5 of 5) - [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025) - [ClimbingCap: Multi-Modal Dataset and Method for Rock Climbing in World Coordinate](https://mlsanity.com/paper/2503.21268/) (2025) - [Triple Spectral Fusion for Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2605.02743/) (2026) - [B'ezier Degradation Modeling for LiDAR-based Human Motion Capture](https://mlsanity.com/paper/2605.19620/) (2026) - [MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations](https://mlsanity.com/paper/2606.23000/) (2026) ## Abstract (arXiv) We introduce HiSC4D, a novel Human-centered interaction and 4D Scene Capture method, aimed at accurately and efficiently creating a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, rich human-human interactions, and human-environment interactions. By utilizing body-mounted IMUs and a head-mounted LiDAR, HiSC4D can capture egocentric human motions in unconstrained space without the need for external devices and pre-built maps. This affords great flexibility and accessibility for human-centered interaction and 4D scene capturing in various environments. Taking into account that IMUs can capture human spatially unrestricted poses but are prone to drifting for long-period using, and while LiDAR is stable for global localization but rough for local positions and orientations, HiSC4D employs a joint optimization method, harmonizing all sensors and utilizing environment cues, yielding promising results for long-term capture in large scenes. To promote research of egocentric human interaction in large scenes and facilitate downstream tasks, we also present a dataset, containing 8 sequences in 4 large scenes (200 to 5,000 m2), providing 36k frames of accurate 4D human motions with SMPL annotations and dynamic scenes, 31k frames of cropped human point clouds, and scene mesh of the environment. A variety of scenarios, such as the basketball gym and commercial street, alongside challenging human motions, such as daily greeting, one-on-one basketball playing, and tour guiding, demonstrate the effectiveness and the generalization ability of HiSC4D. The dataset and code will be publicated on www.lidarhumanmotion.net/hisc4d available for research purposes.