# HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving

arXiv: https://arxiv.org/abs/2212.07729 · HTML: https://mlsanity.com/paper/2212.07729/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [BodyNet: Volumetric Inference of 3D Human Body Shapes](https://mlsanity.com/paper/1804.04875/) (2018)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019)
- [Exploiting temporal context for 3D human pose estimation in the wild](https://mlsanity.com/paper/1905.04266/) (2019)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [Hierarchical Kinematic Human Mesh Recovery](https://mlsanity.com/paper/2003.04232/) (2020)
- [Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild](https://mlsanity.com/paper/2003.07581/) (2020)
- [3D Human Mesh Regression with Dense Correspondence](https://mlsanity.com/paper/2006.05734/) (2020)
- [Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2006.01423/) (2020)
- [3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data](https://mlsanity.com/paper/2011.00980/) (2020)
- [THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers](https://mlsanity.com/paper/2106.09336/) (2021)
- [Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving](https://mlsanity.com/paper/2112.12141/) (2021)
- [PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds](https://mlsanity.com/paper/2003.09644/) (2020)

## Built on by (most cited first, 22 of 22)

- [Robots That Can See: Leveraging Human Pose for Trajectory Prediction](https://mlsanity.com/paper/2309.17209/) (2023)
- [FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models](https://mlsanity.com/paper/2405.05216/) (2024)
- [LiDAR-HMR: 3D Human Mesh Recovery from LiDAR](https://mlsanity.com/paper/2311.11971/) (2023)
- [3D Human Keypoints Estimation From Point Clouds in the Wild Without Human Labels](https://mlsanity.com/paper/2306.04745/) (2023)
- [LPFormer: LiDAR Pose Estimation Transformer with Multi-Task Network](https://mlsanity.com/paper/2306.12525/) (2023)
- [Weakly Supervised Multi-Modal 3D Human Body Pose Estimation for Autonomous Driving](https://mlsanity.com/paper/2307.14889/) (2023)
- [Human Modelling and Pose Estimation Overview](https://mlsanity.com/paper/2406.19290/) (2024)
- [HFGaussian: Learning Generalizable Gaussian Human with Integrated Human Features](https://mlsanity.com/paper/2411.03086/) (2024)
- [Di2Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimation](https://mlsanity.com/paper/2405.17016/) (2024)
- [Towards Multi-Modal Animal Pose Estimation: A Survey and In-Depth Analysis](https://mlsanity.com/paper/2410.09312/) (2024)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [VoxelKP: A Voxel-based Network Architecture for Human Keypoint Estimation in LiDAR Data](https://mlsanity.com/paper/2312.08871/) (2023)
- [Multi-hop graph transformer network for 3D human pose estimation](https://mlsanity.com/paper/2405.03055/) (2024)
- [SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapes](https://mlsanity.com/paper/2408.02291/) (2024)
- [Robust Low-Light Human Pose Estimation through Illumination-Texture Modulation](https://mlsanity.com/paper/2501.08038/) (2025)
- [StarPose: 3D Human Pose Estimation via Spatial-Temporal Autoregressive Diffusion](https://mlsanity.com/paper/2508.02056/) (2025)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [DOR3D-Net: Dense Ordinal Regression Network for 3D Hand Pose Estimation](https://mlsanity.com/paper/2403.13405/) (2024)
- [VOccl3D: A Video Benchmark Dataset for 3D Human Pose and Shape Estimation under real Occlusions](https://mlsanity.com/paper/2508.06757/) (2025)
- [3D Human Pose and Shape Estimation from LiDAR Point Clouds: A Review](https://mlsanity.com/paper/2509.12197/) (2025)
- [Point2Pose: A Generative Framework for 3D Human Pose Estimation with Multi-View Point Cloud Dataset](https://mlsanity.com/paper/2512.10321/) (2025)
- [Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds](https://mlsanity.com/paper/2603.16343/) (2026)

## Abstract (arXiv)

Autonomous driving is an exciting new industry, posing important research questions. Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. While hardware systems and sensors have dramatically improved over the decades -- with cars potentially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information -- not much work has been done to harness these novel signals for the core problem of 3D human pose estimation. Our method, which we coin HUM3DIL (HUMan 3D from Images and LiDAR), efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. It is a fast and compact model for onboard deployment. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages. Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-the-art results on the task of 3D pose estimation.