# D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation

arXiv: https://arxiv.org/abs/2409.14365 · HTML: https://mlsanity.com/paper/2409.14365/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2024.

## Built on by (most cited first, 14 of 14)

- [GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data](https://mlsanity.com/paper/2505.03233/) (2025)
- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [Masked Depth Modeling for Spatial Perception](https://mlsanity.com/paper/2601.17895/) (2026)
- [Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots](https://mlsanity.com/paper/2509.02530/) (2025)
- [Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation](https://mlsanity.com/paper/2512.23705/) (2025)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2510.09320/) (2025)
- [GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation](https://mlsanity.com/paper/2411.18276/) (2024)
- [ExStereo: Lifting 2D Vision-Language-Action Models to 3D with Explicit Stereo Representations](https://mlsanity.com/paper/2610.04805/) (2026)
- [DidSee: Diffusion-Based Depth Completion for Material-Agnostic Robotic Perception and Manipulation](https://mlsanity.com/paper/2506.21034/) (2025)
- [Monocular Depth Estimation From the Perspective of Feature Restoration: A Diffusion Enhanced Depth Restoration Approach](https://mlsanity.com/paper/2604.07664/) (2026)
- [AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior](https://mlsanity.com/paper/2606.25503/) (2026)
- [STEREOFLOW: Progressive Stereo Matching with StereoDiT and Transition Flow Matching](https://mlsanity.com/paper/2607.19986/) (2026)
- [StereoEngine: Scaling Stereo Bimanual Robot Data without Stereo Cameras](https://mlsanity.com/paper/2610.12734/) (2026)

## Abstract (arXiv)

Depth sensing is an important problem for 3D vision-based robotics. Yet, a real-world active stereo or ToF depth camera often produces noisy and incomplete depth which bottlenecks robot performances. In this work, we propose D3RoMa, a learning-based depth estimation framework on stereo image pairs that predicts clean and accurate depth in diverse indoor scenes, even in the most challenging scenarios with translucent or specular surfaces where classical depth sensing completely fails. Key to our method is that we unify depth estimation and restoration into an image-to-image translation problem by predicting the disparity map with a denoising diffusion probabilistic model. At inference time, we further incorporated a left-right consistency constraint as classifier guidance to the diffusion process. Our framework combines recently advanced learning-based approaches and geometric constraints from traditional stereo vision. For model training, we create a large scene-level synthetic dataset with diverse transparent and specular objects to compensate for existing tabletop datasets. The trained model can be directly applied to real-world in-the-wild scenes and achieve state-of-the-art performance in multiple public depth estimation benchmarks. Further experiments in real environments show that accurate depth prediction significantly improves robotic manipulation in various scenarios.