# Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes

arXiv: https://arxiv.org/abs/2308.00628 · HTML: https://mlsanity.com/paper/2308.00628/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views](https://mlsanity.com/paper/1901.04111/) (2019)
- [Direct Multi-view Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2111.04076/) (2021)
- [AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics](https://mlsanity.com/paper/2008.08063/) (2020)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments](https://mlsanity.com/paper/2303.09095/) (2023)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo](https://mlsanity.com/paper/2104.02273/) (2021)
- [LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds](https://mlsanity.com/paper/2203.14698/) (2022)
- [PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections](https://mlsanity.com/paper/1809.03605/) (2018)
- [DOPE: Distillation Of Part Experts for whole-body 3D pose estimation in the wild](https://mlsanity.com/paper/2008.09457/) (2020)
- [FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras](https://mlsanity.com/paper/1610.09534/) (2016)
- [A Flexible Multi-view Multi-modal Imaging System for Outdoor Scenes](https://mlsanity.com/paper/2302.10465/) (2023)

## Built on by (most cited first, 11 of 11)

- [LiDAR-HMR: 3D Human Mesh Recovery from LiDAR](https://mlsanity.com/paper/2311.11971/) (2023)
- [Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation](https://mlsanity.com/paper/2412.13454/) (2024)
- [Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset](https://mlsanity.com/paper/2412.13569/) (2024)
- [FreeCap: Hybrid Calibration-Free Motion Capture in Open Environments](https://mlsanity.com/paper/2411.04469/) (2024)
- [Benchmarking 3D Human Pose Estimation Models under Occlusions](https://mlsanity.com/paper/2504.10350/) (2025)
- [ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening](https://mlsanity.com/paper/2411.09435/) (2024)
- [MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots](https://mlsanity.com/paper/2511.16949/) (2025)
- [3D Human Pose and Shape Estimation from LiDAR Point Clouds: A Review](https://mlsanity.com/paper/2509.12197/) (2025)
- [Mesh-Aware Epipolar Matching for Multi-View Multi-Person 3D Pose Estimation in Basketball](https://mlsanity.com/paper/2605.29953/) (2026)
- [Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration](https://mlsanity.com/paper/2608.02285/) (2026)
- [HuLiGen: Human LiDAR Generation from Parametric Body Models](https://mlsanity.com/paper/2610.10196/) (2026)

## Abstract (arXiv)

3D human pose estimation in outdoor environments has garnered increasing attention recently. However, prevalent 3D human pose datasets pertaining to outdoor scenes lack diversity, as they predominantly utilize only one type of modality (RGB image or pointcloud), and often feature only one individual within each scene. This limited scope of dataset infrastructure considerably hinders the variability of available data. In this article, we propose Human-M3, an outdoor multi-modal multi-view multi-person human pose database which includes not only multi-view RGB videos of outdoor scenes but also corresponding pointclouds. In order to obtain accurate human poses, we propose an algorithm based on multi-modal data input to generate ground truth annotation. This benefits from robust pointcloud detection and tracking, which solves the problem of inaccurate human localization and matching ambiguity that may exist in previous multi-view RGB videos in outdoor multi-person scenes, and generates reliable ground truth annotations. Evaluation of multiple different modalities algorithms has shown that this database is challenging and suitable for future research. Furthermore, we propose a 3D human pose estimation algorithm based on multi-modal data input, which demonstrates the advantages of multi-modal data input for 3D human pose estimation. Code and data will be released on https://github.com/soullessrobot/Human-M3-Dataset.