# Visual Robotic Manipulation with Depth-Aware Pretraining

arXiv: https://arxiv.org/abs/2401.09038 · HTML: https://mlsanity.com/paper/2401.09038/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2024.

## Built on by (most cited first, 12 of 12)

- [Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation](https://mlsanity.com/paper/2411.18623/) (2024)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [DCPI-Depth: Explicitly Infusing Dense Correspondence Prior to Unsupervised Monocular Depth Estimation](https://mlsanity.com/paper/2405.16960/) (2024)
- [Depth Anything at Any Condition](https://mlsanity.com/paper/2507.01634/) (2025)
- [GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation](https://mlsanity.com/paper/2509.15733/) (2025)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation](https://mlsanity.com/paper/2509.19454/) (2025)
- [SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification](https://mlsanity.com/paper/2603.19547/) (2026)
- [KineDepth: Utilizing Robot Kinematics for Online Metric Depth Estimation](https://mlsanity.com/paper/2409.19490/) (2024)
- [Contrastive Action-Image Pre-training for Visuomotor Control](https://mlsanity.com/paper/2606.17256/) (2026)
- [Latent evolving World Action Model](https://mlsanity.com/paper/2609.27455/) (2026)
- [Policy-Calibrated DAgger: Offline Calibrated Noise Injection for Imitation Learning](https://mlsanity.com/paper/2609.30462/) (2026)

## Abstract (arXiv)

Recent work on visual representation learning has shown to be efficient for robotic manipulation tasks. However, most existing works pretrained the visual backbone solely on 2D images or egocentric videos, ignoring the fact that robots learn to act in 3D space, which is hard to learn from 2D observation. In this paper, we examine the effectiveness of pretraining for vision backbone with public-available large-scale 3D data to improve manipulation policy learning. Our method, namely Depth-aware Pretraining for Robotics (DPR), enables an RGB-only backbone to learn 3D scene representations from self-supervised contrastive learning, where depth information serves as auxiliary knowledge. No 3D information is necessary during manipulation policy learning and inference, making our model enjoy both efficiency and effectiveness in 3D space manipulation. Furthermore, we introduce a new way to inject robots' proprioception into the policy networks that makes the manipulation model robust and generalizable. We demonstrate in experiments that our proposed framework improves performance on unseen objects and visual environments for various robotics tasks on both simulated and real robots.