# Seeing and Seeing Through the Glass: Real and Synthetic Data for Multi-Layer Depth Estimation

arXiv: https://arxiv.org/abs/2503.11633 · HTML: https://mlsanity.com/paper/2503.11633/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 43)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [DIODE: A Dense Indoor and Outdoor DEpth Dataset](https://mlsanity.com/paper/1908.00463/) (2019)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)

## Built on by (most cited first, 8 of 8)

- [Sharp Monocular View Synthesis in Less Than a Second](https://mlsanity.com/paper/2512.10685/) (2025)
- [DepthFocus: Controllable Depth Estimation for See-Through Scenes](https://mlsanity.com/paper/2511.16993/) (2025)
- [Modeling Depth Ambiguity: A Mixture-Density Representation for Flying-Point-Free Depth Estimation](https://mlsanity.com/paper/2606.02552/) (2026)
- [The Garbage Dataset (GD): A Multi-Class Image Benchmark for Automated Waste Segregation](https://mlsanity.com/paper/2602.10500/) (2026)
- [ProcFunc: Function-Oriented Abstractions for Procedural 3D Generation in Python](https://mlsanity.com/paper/2604.26943/) (2026)
- [SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping](https://mlsanity.com/paper/2605.28735/) (2026)
- [One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models](https://mlsanity.com/paper/2606.29600/) (2026)
- [GIFT: Geometry-Invariant Fine-Tuning for Non-Lambertian Monocular Depth Estimation](https://mlsanity.com/paper/2608.02068/) (2026)

## Abstract (arXiv)

Transparent objects are common in daily life, and understanding their multi-layer depth information -- perceiving both the transparent surface and the objects behind it -- is crucial for real-world applications that interact with transparent materials. In this paper, we introduce LayeredDepth, the first dataset with multi-layer depth annotations, including a real-world benchmark and a synthetic data generator, to support the task of multi-layer depth estimation. Our real-world benchmark consists of 1,500 images from diverse scenes, and evaluating state-of-the-art depth estimation methods on it reveals that they struggle with transparent objects. The synthetic data generator is fully procedural and capable of providing training data for this task with an unlimited variety of objects and scene compositions. Using this generator, we create a synthetic dataset with 15,300 images. Baseline models training solely on this synthetic dataset produce good cross-domain multi-layer depth estimation. Fine-tuning state-of-the-art single-layer depth models on it substantially improves their performance on transparent objects, with quadruplet accuracy on our benchmark increased from 55.14% to 75.20%. All images and validation annotations are available under CC0 at https://layereddepth.cs.princeton.edu.