# DidSee: Diffusion-Based Depth Completion for Material-Agnostic Robotic Perception and Manipulation

arXiv: https://arxiv.org/abs/2506.21034 · HTML: https://mlsanity.com/paper/2506.21034/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [Efficient Diffusion Training via Min-SNR Weighting Strategy](https://mlsanity.com/paper/2303.09556/) (2023)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Elucidating the Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2308.15321/) (2023)
- [Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think](https://mlsanity.com/paper/2409.11355/) (2024)
- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?](https://mlsanity.com/paper/2403.06090/) (2024)
- [Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps](https://mlsanity.com/paper/2305.15583/) (2023)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking](https://mlsanity.com/paper/2406.04316/) (2024)
- [ClearPose: Large-scale Transparent Object Dataset and Benchmark](https://mlsanity.com/paper/2203.03890/) (2022)
- [Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning](https://mlsanity.com/paper/2402.02500/) (2024)
- [Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects](https://mlsanity.com/paper/2110.00087/) (2021)
- [Exploiting the Signal-Leak Bias in Diffusion Models](https://mlsanity.com/paper/2309.15842/) (2023)
- [Addressing Negative Transfer in Diffusion Models](https://mlsanity.com/paper/2306.00354/) (2023)
- [Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D](https://mlsanity.com/paper/2401.12046/) (2024)
- [On Error Propagation of Diffusion Models](https://mlsanity.com/paper/2308.05021/) (2023)
- [Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2403.09176/) (2024)
- [Robotic Perception of Transparent Objects: A Review](https://mlsanity.com/paper/2304.00157/) (2023)
- [D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation](https://mlsanity.com/paper/2409.14365/) (2024)
- [TODE-Trans: Transparent Object Depth Estimation with Transformer](https://mlsanity.com/paper/2209.08455/) (2022)
- [FDCT: Fast Depth Completion for Transparent Objects](https://mlsanity.com/paper/2307.12274/) (2023)
- [LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow](https://mlsanity.com/paper/2409.05688/) (2024)
- [Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation](https://mlsanity.com/paper/2410.21708/) (2024)

## Abstract (arXiv)

Commercial RGB-D cameras often produce noisy, incomplete depth maps for non-Lambertian objects. Traditional depth completion methods struggle to generalize due to the limited diversity and scale of training data. Recent advances exploit visual priors from pre-trained text-to-image diffusion models to enhance generalization in dense prediction tasks. However, we find that biases arising from training-inference mismatches in the vanilla diffusion framework significantly impair depth completion performance. Additionally, the lack of distinct visual features in non-Lambertian regions further hinders precise prediction. To address these issues, we propose DidSee, a diffusion-based framework for depth completion on non-Lambertian objects. First, we integrate a rescaled noise scheduler enforcing a zero terminal signal-to-noise ratio to eliminate signal leakage bias. Second, we devise a noise-agnostic single-step training formulation to alleviate error accumulation caused by exposure bias and optimize the model with a task-specific loss. Finally, we incorporate a semantic enhancer that enables joint depth completion and semantic segmentation, distinguishing objects from backgrounds and yielding precise, fine-grained depth maps. DidSee achieves state-of-the-art performance on multiple benchmarks, demonstrates robust real-world generalization, and effectively improves downstream tasks such as category-level pose estimation and robotic grasping.