# PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition

arXiv: https://arxiv.org/abs/1611.06448 · HTML: https://mlsanity.com/paper/1611.06448/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2016.

## Built on by (most cited first, 18 of 18)

- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network](https://mlsanity.com/paper/2207.10434/) (2022)
- [Human-Aided Saliency Maps Improve Generalization of Deep Learning](https://mlsanity.com/paper/2105.03492/) (2021)
- [Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention](https://mlsanity.com/paper/2303.01542/) (2023)
- [Fair SA: Sensitivity Analysis for Fairness in Face Recognition](https://mlsanity.com/paper/2202.03586/) (2022)
- [Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty](https://mlsanity.com/paper/2404.10980/) (2024)
- [Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts](https://mlsanity.com/paper/2107.12052/) (2021)
- [Motif Mining: Finding and Summarizing Remixed Image Content](https://mlsanity.com/paper/2203.08327/) (2022)
- [Measuring Human Perception to Improve Open Set Recognition](https://mlsanity.com/paper/2209.03519/) (2022)
- [Using Human Perception to Regularize Transfer Learning](https://mlsanity.com/paper/2211.07885/) (2022)
- [NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency Response Scenarios](https://mlsanity.com/paper/2309.09518/) (2023)
- [Guiding Machine Perception with Psychophysics](https://mlsanity.com/paper/2207.02241/) (2022)
- [Out-of-Distribution Example Detection in Deep Neural Networks using Distance to Modelled Embedding](https://mlsanity.com/paper/2108.10673/) (2021)
- [A high-precision self-supervised monocular visual odometry in foggy weather based on robust cycled generative adversarial networks and multi-task learning aided depth estimation](https://mlsanity.com/paper/2203.04812/) (2022)
- [Psychophysical-Score: A Behavioral Measure for Assessing the Biological Plausibility of Visual Recognition Models](https://mlsanity.com/paper/2210.08632/) (2022)
- [Explaining Deep Face Algorithms through Visualization: A Survey](https://mlsanity.com/paper/2309.14715/) (2023)
- [Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue](https://mlsanity.com/paper/2412.05553/) (2024)
- [HiLLIE: Human-in-the-Loop Training for Low-Light Image Enhancement](https://mlsanity.com/paper/2505.02134/) (2025)

## Abstract (arXiv)

By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies.