# How Aligned are Different Alignment Metrics?

arXiv: https://arxiv.org/abs/2407.07530 · HTML: https://mlsanity.com/paper/2407.07530/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency](https://mlsanity.com/paper/2006.16736/) (2020)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Alignment with human representations supports robust few-shot learning](https://mlsanity.com/paper/2301.11990/) (2023)
- [Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex](https://mlsanity.com/paper/2306.03779/) (2023)
- [Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?](https://mlsanity.com/paper/2305.17023/) (2023)

## Built on by (most cited first, 6 of 6)

- [Representational Similarity via Interpretable Visual Concepts](https://mlsanity.com/paper/2503.15699/) (2025)
- [Alignment and Adversarial Robustness: Are More Human-Like Models More Secure?](https://mlsanity.com/paper/2502.12377/) (2025)
- [Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks](https://mlsanity.com/paper/2409.06362/) (2024)
- [Evaluating Imputation Techniques for Short-Term Gaps in Heart Rate Data](https://mlsanity.com/paper/2508.08268/) (2025)
- [CLPIPS: A Personalized Metric for AI-Generated Image Similarity](https://mlsanity.com/paper/2604.01234/) (2026)
- [Capability neq Interpretability: Human Interpretability of Vision Foundation Models](https://mlsanity.com/paper/2605.20337/) (2026)

## Abstract (arXiv)

In recent years, various methods and benchmarks have been proposed to empirically evaluate the alignment of artificial neural networks to human neural and behavioral data. But how aligned are different alignment metrics? To answer this question, we analyze visual data from Brain-Score (Schrimpf et al., 2018), including metrics from the model-vs-human toolbox (Geirhos et al., 2021), together with human feature alignment (Linsley et al., 2018; Fel et al., 2022) and human similarity judgements (Muttenthaler et al., 2022). We find that pairwise correlations between neural scores and behavioral scores are quite low and sometimes even negative. For instance, the average correlation between those 80 models on Brain-Score that were fully evaluated on all 69 alignment metrics we considered is only 0.198. Assuming that all of the employed metrics are sound, this implies that alignment with human perception may best be thought of as a multidimensional concept, with different methods measuring fundamentally different aspects. Our results underline the importance of integrative benchmarking, but also raise questions about how to correctly combine and aggregate individual metrics. Aggregating by taking the arithmetic average, as done in Brain-Score, leads to the overall performance currently being dominated by behavior (95.25% explained variance) while the neural predictivity plays a less important role (only 33.33% explained variance). As a first step towards making sure that different alignment metrics all contribute fairly towards an integrative benchmark score, we therefore conclude by comparing three different aggregation options.