# Mobile AR Depth Estimation: Challenges & Prospects -- Extended Version

arXiv: https://arxiv.org/abs/2310.14437 · HTML: https://mlsanity.com/paper/2310.14437/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth](https://mlsanity.com/paper/1904.02028/) (2019)
- [DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenes](https://mlsanity.com/paper/2110.11590/) (2021)
- [Toward Practical Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2112.02306/) (2021)
- [Focus on defocus: bridging the synthetic to real domain gap for depth estimation](https://mlsanity.com/paper/2005.09623/) (2020)
- [Deep Depth from Focal Stack with Defocus Model for Camera-Setting Invariance](https://mlsanity.com/paper/2202.13055/) (2022)

## Built on by (most cited first, 3 of 3)

- [Generative Augmented Reality: Paradigms, Technologies, and Future Applications](https://mlsanity.com/paper/2511.16783/) (2025)
- [GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos](https://mlsanity.com/paper/2412.02267/) (2024)
- [VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recovery](https://mlsanity.com/paper/2505.02704/) (2025)

## Abstract (arXiv)

Metric depth estimation plays an important role in mobile augmented reality (AR). With accurate metric depth, we can achieve more realistic user interactions such as object placement and occlusion detection. While specialized hardware like LiDAR demonstrates its promise, its restricted availability, i.e., only on selected high-end mobile devices, and performance limitations such as range and sensitivity to the environment, make it less ideal. Monocular depth estimation, on the other hand, relies solely on mobile cameras, which are ubiquitous, making it a promising alternative for mobile AR. In this paper, we investigate the challenges and opportunities of achieving accurate metric depth estimation in mobile AR. We tested four different state-of-the-art monocular depth estimation models on a newly introduced dataset (ARKitScenes) and identified three types of challenges: hard-ware, data, and model related challenges. Furthermore, our research provides promising future directions to explore and solve those challenges. These directions include (i) using more hardware-related information from the mobile device's camera and other available sensors, (ii) capturing high-quality data to reflect real-world AR scenarios, and (iii) designing a model architecture to utilize the new information.