# Weakly Supervised Multi-Modal 3D Human Body Pose Estimation for Autonomous Driving

arXiv: https://arxiv.org/abs/2307.14889 · HTML: https://mlsanity.com/paper/2307.14889/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Semantic Graph Convolutional Networks for 3D Human Pose Regression](https://mlsanity.com/paper/1904.03345/) (2019)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [The EuroCity Persons Dataset: A Novel Benchmark for Object Detection](https://mlsanity.com/paper/1805.07193/) (2018)
- [Can 3D Pose be Learned from 2D Projections Alone?](https://mlsanity.com/paper/1808.07182/) (2018)
- [MotionMixer: MLP-based 3D Human Body Pose Forecasting](https://mlsanity.com/paper/2207.00499/) (2022)
- [Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving](https://mlsanity.com/paper/2112.12141/) (2021)
- [PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections](https://mlsanity.com/paper/1809.03605/) (2018)
- [Traffic Control Gesture Recognition for Autonomous Vehicles](https://mlsanity.com/paper/2007.16072/) (2020)
- [HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving](https://mlsanity.com/paper/2212.07729/) (2022)
- [Self-Supervised 3D Human Pose Estimation with Multiple-View Geometry](https://mlsanity.com/paper/2108.07777/) (2021)
- [On-Board Pedestrian Trajectory Prediction Using Behavioral Features](https://mlsanity.com/paper/2210.11999/) (2022)
- [Learning Temporal 3D Human Pose Estimation with Pseudo-Labels](https://mlsanity.com/paper/2110.07578/) (2021)

## Built on by (most cited first, 8 of 8)

- [MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network](https://mlsanity.com/paper/2310.16288/) (2023)
- [Optimizing Local-Global Dependencies for Accurate 3D Human Pose Estimation](https://mlsanity.com/paper/2412.19676/) (2024)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [Towards a large-scale fused and labeled dataset of human pose while interacting with robots in shared urban areas](https://mlsanity.com/paper/2402.10077/) (2024)
- [KASportsFormer: Kinematic Anatomy Enhanced Transformer for 3D Human Pose Estimation on Short Sports Scene Video](https://mlsanity.com/paper/2507.20763/) (2025)
- [VOccl3D: A Video Benchmark Dataset for 3D Human Pose and Shape Estimation under real Occlusions](https://mlsanity.com/paper/2508.06757/) (2025)
- [3D Human Pose and Shape Estimation from LiDAR Point Clouds: A Review](https://mlsanity.com/paper/2509.12197/) (2025)
- [Point2Pose: A Generative Framework for 3D Human Pose Estimation with Multi-View Point Cloud Dataset](https://mlsanity.com/paper/2512.10321/) (2025)

## Abstract (arXiv)

Accurate 3D human pose estimation (3D HPE) is crucial for enabling autonomous vehicles (AVs) to make informed decisions and respond proactively in critical road scenarios. Promising results of 3D HPE have been gained in several domains such as human-computer interaction, robotics, sports and medical analytics, often based on data collected in well-controlled laboratory environments. Nevertheless, the transfer of 3D HPE methods to AVs has received limited research attention, due to the challenges posed by obtaining accurate 3D pose annotations and the limited suitability of data from other domains. We present a simple yet efficient weakly supervised approach for 3D HPE in the AV context by employing a high-level sensor fusion between camera and LiDAR data. The weakly supervised setting enables training on the target datasets without any 2D/3D keypoint labels by using an off-the-shelf 2D joint extractor and pseudo labels generated from LiDAR to image projections. Our approach outperforms state-of-the-art results by up to ~ 13% on the Waymo Open Dataset in the weakly supervised setting and achieves state-of-the-art results in the supervised setting.