# Mirror3D: Depth Refinement for Mirror Surfaces

arXiv: https://arxiv.org/abs/2106.06629 · HTML: https://mlsanity.com/paper/2106.06629/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)
- [Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks](https://mlsanity.com/paper/1605.02305/) (2016)
- [PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1804.06278/) (2018)
- [Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding](https://mlsanity.com/paper/1902.09777/) (2019)
- [StereoDRNet: Dilated Residual Stereo Net](https://mlsanity.com/paper/1904.02251/) (2019)
- [Indoor Depth Completion with Boundary Consistency and Self-Attention](https://mlsanity.com/paper/1908.08344/) (2019)
- [On Deep Learning Techniques to Boost Monocular Depth Estimation for Autonomous Navigation](https://mlsanity.com/paper/2010.06626/) (2020)
- [Multi-modal Transfer Learning for Grasping Transparent and Specular Objects](https://mlsanity.com/paper/2006.00028/) (2020)
- [Fine-To-Coarse Global Registration of RGB-D Scans](https://mlsanity.com/paper/1607.08539/) (2016)
- [Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects](https://mlsanity.com/paper/1409.4095/) (2014)

## Built on by (most cited first, 13 of 13)

- [Glass Segmentation with RGB-Thermal Image Pairs](https://mlsanity.com/paper/2204.05453/) (2022)
- [Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-supervised Learning](https://mlsanity.com/paper/2109.10563/) (2021)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces](https://mlsanity.com/paper/2408.06083/) (2024)
- [Diffusion-Based Depth Inpainting for Transparent and Reflective Objects](https://mlsanity.com/paper/2410.08567/) (2024)
- [MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World](https://mlsanity.com/paper/2504.15397/) (2025)
- [Deep Learning in Concealed Dense Prediction](https://mlsanity.com/paper/2504.10979/) (2025)
- [3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data](https://mlsanity.com/paper/2403.06538/) (2024)
- [Reflect3r: Single-View 3D Stereo Reconstruction Aided by Mirror Reflections](https://mlsanity.com/paper/2509.20607/) (2025)
- [DEPTHOR++: Robust Depth Enhancement from a Real-World Lightweight dToF and RGB Guidance](https://mlsanity.com/paper/2509.26498/) (2025)
- [DEUX: Active Exploration for Learning Unsupervised Depth Perception](https://mlsanity.com/paper/2310.06164/) (2023)
- [Fusion of Short-term and Long-term Attention for Video Mirror Detection](https://mlsanity.com/paper/2407.07999/) (2024)
- [GIFT: Geometry-Invariant Fine-Tuning for Non-Lambertian Monocular Depth Estimation](https://mlsanity.com/paper/2608.02068/) (2026)

## Abstract (arXiv)

Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors. To address this problem, we create the Mirror3D dataset: a 3D mirror plane dataset based on three RGBD datasets (Matterport3D, NYUv2 and ScanNet) containing 7,011 mirror instance masks and 3D planes. We then develop Mirror3DNet: a module that refines raw sensor depth or estimated depth to correct errors on mirror surfaces. Our key idea is to estimate the 3D mirror plane based on RGB input and surrounding depth context, and use this estimate to directly regress mirror surface depth. Our experiments show that Mirror3DNet significantly mitigates errors from a variety of input depth data, including raw sensor depth and depth estimation or completion methods.