mlsanityData to 7 Oct 2026

Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving

arXiv 2112.12141 ↗First posted 22 Dec 2021cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

3D human pose estimation (HPE) in autonomous vehicles (AV) differs from other use cases in many factors, including the 3D resolution and range of data, absence of dense depth maps, failure modes for LiDAR, relative location between the camera and LiDAR, and a high bar for estimation accuracy. Data collected for other use cases (such as virtual reality, gaming, and animation) may therefore not be usable for AV applications. This necessitates the collection and annotation of a large amount of 3D data for HPE in AV, which is time-consuming and expensive. In this paper, we propose one of the first approaches to alleviate this problem in the AV setting. Specifically, we propose a multi-modal approach which uses 2D labels on RGB images as weak supervision to perform 3D HPE. The proposed multi-modal architecture incorporates LiDAR and camera inputs with an auxiliary segmentation branch. On the Waymo Open Dataset, our approach achieves a 22% relative improvement over camera-only 2D HPE baseline, and 6% improvement over LiDAR-only model. Finally, careful ablation studies and parts based analysis illustrate the advantages of each of our contributions.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 22 Dec 2021.

Builds on

36 of 51 references.

References · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  6. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  7. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  8. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  9. 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
  10. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  11. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  12. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
24 more
  1. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 205
  2. 2017Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approachcited by 114
  3. 20163D Human Pose Estimation = 2D Pose Estimation + Matchingcited by 112
  4. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  5. 2018Ordinal Depth Supervision for 3D Human Pose Estimationcited by 107
  6. 2019Self-Supervised Learning of 3D Human Pose using Multi-view Geometrycited by 98
  7. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  8. 2020SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approachcited by 94
  9. 2017Exploiting temporal information for 3D pose estimationcited by 89
  10. 20183D Human Pose Estimation in the Wild by Adversarial Learningcited by 86
  11. 2019RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimationcited by 85
  12. 2020Cascaded deep monocular 3D human pose estimation with evolutionary training datacited by 80
  13. 2017Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Imagecited by 73
  14. 2018Learning Monocular 3D Human Pose Estimation from Multi-view Imagescited by 70
  15. 2018Photo Wake-Up: 3D Character Animation from a Single Photocited by 60
  16. 2020VPN: Learning Video-Pose Embedding for Activities of Daily Livingcited by 57
  17. 2015Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Videocited by 54
  18. 20203D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Trainingcited by 54
  19. 2020AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wildcited by 40
  20. 20183D Human Pose Estimation in RGBD Images for Robotic Task Learningcited by 33
  21. 2020Vid2Player: Controllable Video Sprites that Behave and Appear like Professional Tennis Playerscited by 30
  22. 2014Robust Estimation of 3D Human Poses from a Single Imagecited by 23
  23. 2020HPERL: 3D Human Pose Estimation from RGB and LiDARcited by 18
  24. 2020PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 12; 2025: 9; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensingcited by 77
  2. 2024Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioningcited by 29
  3. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  4. 2022Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDARcited by 23
  5. 2024FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Modelscited by 23
  6. 2022HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Drivingcited by 22
  7. 2024RELI11D: A Comprehensive Multimodal Human Motion Dataset and Methodcited by 16
  8. 2023Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypointscited by 12
  9. 2023LiDAR-HMR: 3D Human Mesh Recovery from LiDARcited by 12
  10. 2024X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensingcited by 12
  11. 2025Towards Balanced Multi-Modal Learning in 3D Human Pose Estimationcited by 11
  12. 20233D Human Keypoints Estimation From Point Clouds in the Wild Without Human Labelscited by 9
27 more
  1. 2023LPFormer: LiDAR Pose Estimation Transformer with Multi-Task Networkcited by 9
  2. 2023Weakly Supervised Multi-Modal 3D Human Body Pose Estimation for Autonomous Drivingcited by 8
  3. 2025Towards Robust and Realistic Human Pose Estimation via WiFi Signalscited by 5
  4. 2024Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimationcited by 4
  5. 2022MUG: Multi-human Graph Network for 3D Mesh Reconstruction from 2D Posecited by 3
  6. 2023BoIR: Box-Supervised Instance Representation for Multi-Person Pose Estimationcited by 2
  7. 2024Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensorscited by 2
  8. 2024STGFormer: Spatio-Temporal GraphFormer for 3D Human Pose Estimation in Videocited by 2
  9. 2024Towards Multi-Modal Animal Pose Estimation: A Survey and In-Depth Analysiscited by 2
  10. 2025DreamPose3D: Hallucinative Diffusion with Prompt Learning for 3D Human Pose Estimationcited by 2
  11. 2023Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuningcited by 1
  12. 2023LiCamPose: Combining Multi-View LiDAR and RGB Cameras for Robust Single-timestamp 3D Human Pose Estimationcited by 1
  13. 2023VoxelKP: A Voxel-based Network Architecture for Human Keypoint Estimation in LiDAR Datacited by 1
  14. 2024PoseAdapt: Sustainable Human Pose Estimation via Continual Learning Benchmarks and Toolkitcited by 1
  15. 2025GenHPE: Generative Counterfactuals for 3D Human Pose Estimation with Radio Frequency Signalscited by 1
  16. 2025HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoningcited by 1
  17. 2025StarPose: 3D Human Pose Estimation via Spatial-Temporal Autoregressive Diffusioncited by 1
  18. 2023E3Pose: Energy-Efficient Edge-assisted Multi-camera System for Multi-human 3D Pose Estimationcited by 0
  19. 2023A Benchmark Generative Probabilistic Model for Weak Supervised Learningcited by 0
  20. 2024Towards a large-scale fused and labeled dataset of human pose while interacting with robots in shared urban areascited by 0
  21. 2024DOR3D-Net: Dense Ordinal Regression Network for 3D Hand Pose Estimationcited by 0
  22. 20253D Human Pose and Shape Estimation from LiDAR Point Clouds: A Reviewcited by 0
  23. 2025SkelSplat: Robust Multi-view 3D Human Pose Estimation with Differentiable Gaussian Renderingcited by 0
  24. 2025Point2Pose: A Generative Framework for 3D Human Pose Estimation with Multi-View Point Cloud Datasetcited by 0
  25. 2026SimpliHuMoN: Simplifying Human Motion Predictioncited by 0
  26. 2026Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Cloudscited by 0
  27. 2026NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimationcited by 0