# Evaluating Perceptual Distance Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data

arXiv: https://arxiv.org/abs/2403.10390 · HTML: https://mlsanity.com/paper/2403.10390/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 6 of 6)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [An Unsupervised Information-Theoretic Perceptual Quality Metric](https://mlsanity.com/paper/2006.06752/) (2020)
- [Iterative Gaussianization: from ICA to Random Rotations](https://mlsanity.com/paper/1602.00229/) (2016)
- [A practical guide and software for analysing pairwise comparison experiments](https://mlsanity.com/paper/1712.03686/) (2017)
- [PerceptNet: A Human Visual System Inspired Neural Network for Estimating Perceptual Distance](https://mlsanity.com/paper/1910.12548/) (2019)

## Abstract (arXiv)

The Two Alternative Forced Choice (2AFC) paradigm offers advantages over the Mean Opinion Score (MOS) paradigm in psychophysics (PF), such as simplicity and robustness. However, when evaluating perceptual distance models, MOS enables direct correlation between model predictions and PF data. In contrast, 2AFC only allows pairwise comparisons to be converted into a quality ranking similar to MOS when comparisons include shared images. In large datasets, like BAPPS, where image patches and distortions are combined randomly, deriving rankings from 2AFC PF data becomes infeasible, as distorted images included in each comparisons are independent. To address this, instead of relying on MOS correlation, researchers have trained ad-hoc neural networks to reproduce 2AFC PF data based on pairs of model distances - a black-box approach with conceptual and operational limitations. This paper introduces a more robust distance-model evaluation method using a pure probabilistic approach, applying maximum likelihood estimation to a binomial decision model. Our method demonstrates superior simplicity, interpretability, flexibility, and computational efficiency, as shown through evaluations of various visual distance models on two 2AFC PF datasets.