# Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB

arXiv: https://arxiv.org/abs/2409.11104 · HTML: https://mlsanity.com/paper/2409.11104/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Exploiting temporal information for 3D pose estimation](https://mlsanity.com/paper/1711.08585/) (2017)
- [3D Human Pose Estimation in the Wild by Adversarial Learning](https://mlsanity.com/paper/1803.09722/) (2018)
- [3D Human Pose Estimation from a Single Image via Distance Matrix Regression](https://mlsanity.com/paper/1611.09010/) (2016)
- [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://mlsanity.com/paper/2201.07436/) (2022)
- [Compressed Volumetric Heatmaps for Multi-Person 3D Pose Estimation](https://mlsanity.com/paper/2004.00329/) (2020)
- [3D Human Pose Estimation in RGBD Images for Robotic Task Learning](https://mlsanity.com/paper/1803.02622/) (2018)
- [Human Pose Estimation in Extremely Low-Light Conditions](https://mlsanity.com/paper/2303.15410/) (2023)
- [3D Human Pose Estimation with 2D Marginal Heatmaps](https://mlsanity.com/paper/1806.01484/) (2018)
- [Kinect Range Sensing: Structured-Light versus Time-of-Flight Kinect](https://mlsanity.com/paper/1505.05459/) (2015)
- [Visible-Infrared Person Re-Identification Using Privileged Intermediate Information](https://mlsanity.com/paper/2209.09348/) (2022)
- [Synthetic Occlusion Augmentation with Volumetric Heatmaps for the 2018 ECCV PoseTrack Challenge on 3D Human Pose Estimation](https://mlsanity.com/paper/1809.04987/) (2018)
- [Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps](https://mlsanity.com/paper/2207.02519/) (2022)

## Built on by (most cited first, 1 of 1)

- [Point2Pose: A Generative Framework for 3D Human Pose Estimation with Multi-View Point Cloud Dataset](https://mlsanity.com/paper/2512.10321/) (2025)

## Abstract (arXiv)

Despite the recent advances in computer vision research, estimating the 3D human pose from single RGB images remains a challenging task, as multiple 3D poses can correspond to the same 2D projection on the image. In this context, depth data could help to disambiguate the 2D information by providing additional constraints about the distance between objects in the scene and the camera. Unfortunately, the acquisition of accurate depth data is limited to indoor spaces and usually is tied to specific depth technologies and devices, thus limiting generalization capabilities. In this paper, we propose a method able to leverage the benefits of depth information without compromising its broader applicability and adaptability in a predominantly RGB-camera-centric landscape. Our approach consists of a heatmap-based 3D pose estimator that, leveraging the paradigm of Privileged Information, is able to hallucinate depth information from the RGB frames given at inference time. More precisely, depth information is used exclusively during training by enforcing our RGB-based hallucination network to learn similar features to a backbone pre-trained only on depth data. This approach proves to be effective even when dealing with limited and small datasets. Experimental results reveal that the paradigm of Privileged Information significantly enhances the model's performance, enabling efficient extraction of depth information by using only RGB images.