# AR as an Evaluation Playground: Bridging Metrics and Visual Perception of Computer Vision Models

arXiv: https://arxiv.org/abs/2508.04102 · HTML: https://mlsanity.com/paper/2508.04102/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Align3R: Aligned Monocular Depth Estimation for Dynamic Videos](https://mlsanity.com/paper/2412.03079/) (2024)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [HDR Environment Map Estimation for Real-Time Augmented Reality](https://mlsanity.com/paper/2011.10687/) (2020)
- [ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation](https://mlsanity.com/paper/2403.18807/) (2024)
- [Focus on defocus: bridging the synthetic to real domain gap for depth estimation](https://mlsanity.com/paper/2005.09623/) (2020)
- [Learning Multi-Arm Manipulation Through Collaborative Teleoperation](https://mlsanity.com/paper/2012.06738/) (2020)
- [Synthetic-to-Real Self-supervised Robust Depth Estimation via Learning with Motion and Structure Priors](https://mlsanity.com/paper/2503.20211/) (2025)
- [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022)
- [ARGUS: Visualization of AI-Assisted Task Guidance in AR](https://mlsanity.com/paper/2308.06246/) (2023)
- [HybridDepth: Robust Metric Depth Fusion by Leveraging Depth from Focus and Single-Image Priors](https://mlsanity.com/paper/2407.18443/) (2024)
- [Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble](https://mlsanity.com/paper/2501.13964/) (2025)
- [Raising the Bar on the Evaluation of Out-of-Distribution Detection](https://mlsanity.com/paper/2209.11960/) (2022)
- [Multimodal Object Detection using Depth and Image Data for Manufacturing Parts](https://mlsanity.com/paper/2411.09062/) (2024)
- [Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality](https://mlsanity.com/paper/2106.15280/) (2021)
- [Towards a Perceptual Evaluation Framework for Lighting Estimation](https://mlsanity.com/paper/2312.04334/) (2023)
- [Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI](https://mlsanity.com/paper/2410.04652/) (2024)
- [TOM: A Development Platform For Wearable Intelligent Assistants](https://mlsanity.com/paper/2407.15523/) (2024)

## Abstract (arXiv)

Quantitative metrics are central to evaluating computer vision (CV) models, but they often fail to capture real-world performance due to protocol inconsistencies and ground-truth noise. While visual perception studies can complement these metrics, they often require end-to-end systems that are time-consuming to implement and setups that are difficult to reproduce. We systematically summarize key challenges in evaluating CV models and present the design of ARCADE, an evaluation platform that leverages augmented reality (AR) to enable easy, reproducible, and human-centered CV evaluation. ARCADE uses a modular architecture that provides cross-platform data collection, pluggable model inference, and interactive AR tasks, supporting both metric and visual perception evaluation. We demonstrate ARCADE through a user study with 15 participants and case studies on two representative CV tasks, depth and lighting estimation, showing that ARCADE can reveal perceptual flaws in model quality that are often missed by traditional metrics. We also evaluate ARCADE's usability and performance, showing its flexibility as a reliable real-time platform.