# Perceived face similarity reflects inverse-generative and naturalistic discriminative objectives

arXiv: https://arxiv.org/abs/2605.12619 · HTML: https://mlsanity.com/paper/2605.12619/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 97 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 29 of 29)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future](https://mlsanity.com/paper/2001.07092/) (2020)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Simple and Effective VAE Training with Calibrated Decoders](https://mlsanity.com/paper/2006.13202/) (2020)
- [Deeply learned face representations are sparse, selective, and robust](https://mlsanity.com/paper/1412.1265/) (2014)
- [Third Time's the Charm? Image and Video Editing with StyleGAN3](https://mlsanity.com/paper/2201.13433/) (2022)
- [YOLO5Face: Why Reinventing a Face Detector](https://mlsanity.com/paper/2105.12931/) (2021)
- [Surpassing Human-Level Face Verification Performance on LFW with GaussianFace](https://mlsanity.com/paper/1404.3840/) (2014)
- [Deep Convolutional Neural Networks in the Face of Caricature: Identity and Image Revealed](https://mlsanity.com/paper/1812.10902/) (2018)
- [Statistical inference on representational geometries](https://mlsanity.com/paper/2112.09200/) (2021)
- [Discriminating image representations with principal distortions](https://mlsanity.com/paper/2410.15433/) (2024)
- [Testing the limits of natural language models for predicting human language judgments](https://mlsanity.com/paper/2204.03592/) (2022)
- [Twin identification over viewpoint change: A deep convolutional neural network surpasses humans](https://mlsanity.com/paper/2207.05316/) (2022)
- [Distinguishing representational geometries with controversial stimuli: Bayesian experimental design and its application to face dissimilarity judgments](https://mlsanity.com/paper/2211.15053/) (2022)
- [Model-Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn't the Right One](https://mlsanity.com/paper/2510.23321/) (2025)
- [Modeling biological face recognition with deep convolutional neural networks](https://mlsanity.com/paper/2208.06681/) (2022)

## Abstract (arXiv)

The perceptual representations supporting our ability to recognize faces remain a computational mystery. Deep neural networks offer mechanistic hypotheses for human face perception, but theoretically distinct models often make indistinguishable representational predictions for randomly sampled faces. To expose diagnostic differences among these hypotheses, we compared six neural network models sharing an architecture but trained on distinct tasks, using face pairs optimized to elicit contrasting model predictions ("controversial" pairs) alongside randomly sampled pairs. We tested model predictions against face-dissimilarity judgments from 864 human participants across stimulus sets differing in realism and pose variation. Models prioritizing high-level, invariant structures (trained via inverse rendering, face identification, or object classification) most robustly matched human judgments. Furthermore, models trained on natural images typically outperformed synthetic-trained counterparts. Together, these findings suggest that human face perception is shaped by mechanisms that infer latent causes of facial appearance, discount nuisance variation, and are tuned by natural image statistics.