# Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding

arXiv: https://arxiv.org/abs/2503.15770 · HTML: https://mlsanity.com/paper/2503.15770/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 77 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [Towards Zero-Shot Scale-Aware Monocular Depth Estimation](https://mlsanity.com/paper/2306.17253/) (2023)
- [Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation](https://mlsanity.com/paper/2412.14015/) (2024)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Learning Single Camera Depth Estimation using Dual-Pixels](https://mlsanity.com/paper/1904.05822/) (2019)
- [Single Image Depth Estimation Trained via Depth from Defocus Cues](https://mlsanity.com/paper/2001.05036/) (2020)
- [Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration](https://mlsanity.com/paper/2012.00301/) (2020)
- [Depth Prompting for Sensor-Agnostic Depth Estimation](https://mlsanity.com/paper/2405.11867/) (2024)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Spatially Varying Nanophotonic Neural Networks](https://mlsanity.com/paper/2308.03407/) (2023)
- [Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging](https://mlsanity.com/paper/2402.18102/) (2024)
- [Joint Image and Depth Estimation with Mask-Based Lensless Cameras](https://mlsanity.com/paper/1910.02526/) (2019)
- [Camera-Independent Single Image Depth Estimation from Defocus Blur](https://mlsanity.com/paper/2311.13045/) (2023)
- [PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing](https://mlsanity.com/paper/2207.00945/) (2022)
- [Depolarized Holography with Polarization-multiplexing Metasurface](https://mlsanity.com/paper/2309.14668/) (2023)
- [Metasurface-enhanced Light Detection and Ranging Technology](https://mlsanity.com/paper/2204.04208/) (2022)

## Abstract (arXiv)

Depth foundation models (DFMs) offer strong learned priors for 3D perception from single RGB images but lack physical depth cues, leading to ambiguities in metric scale. We introduce metalenses, an emerging class of ultrathin planar optical elements, as a solution to physically encode missing metric depth cues via nanophotonics. In this paper, we bridge the gap between metalens and DFMs to achieve accurate metric monocular depth sensing. In a single monocular shot, our metalens embeds depth-dependent positional shifts into two polarized optical wavefronts. With an input adaptation strategty, we enable direct fine-tuning that aligns a pretrained DFM with the optical signals. To scale the training data, we further develop a comprehensive simulation pipeline that synthesizes metalens responses from RGB-D datasets, incorporating physical factors to minimize the sim-to-real gap. Experiments demonstrate that this approach outperforms both monocular metric depth estimation and depth-from-defocus baselines, showing an effective pathway for accurate monocular metric depth sensing.