# Dimensions underlying the representational alignment of deep neural networks with humans

arXiv: https://arxiv.org/abs/2406.19087 · HTML: https://mlsanity.com/paper/2406.19087/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 87 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [On the importance of single directions for generalization](https://mlsanity.com/paper/1803.06959/) (2018)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learning](https://mlsanity.com/paper/1904.11740/) (2019)
- [Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future](https://mlsanity.com/paper/2001.07092/) (2020)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Understanding Neural Networks via Feature Visualization: A survey](https://mlsanity.com/paper/1904.08939/) (2019)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [The neuroconnectionist research programme](https://mlsanity.com/paper/2209.03718/) (2022)
- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [Revisiting the Importance of Individual Units in CNNs via Ablation](https://mlsanity.com/paper/1806.02891/) (2018)
- [Don't trust your eyes: on the (un)reliability of feature visualizations](https://mlsanity.com/paper/2306.04719/) (2023)

## Built on by (most cited first, 22 of 22)

- [Human-like object concept representations emerge naturally in multimodal large language models](https://mlsanity.com/paper/2407.01067/) (2024)
- [Disentangling the Factors of Convergence between Brains and Computer Vision Models](https://mlsanity.com/paper/2508.18226/) (2025)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [Revealing emergent human-like conceptual representations from language prediction](https://mlsanity.com/paper/2501.12547/) (2025)
- [Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era](https://mlsanity.com/paper/2506.03994/) (2025)
- [Geometric Stability: The Missing Axis of Representations](https://mlsanity.com/paper/2601.09173/) (2026)
- [Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Brain](https://mlsanity.com/paper/2601.22722/) (2026)
- [Characterizing Universal Object Representations Across Vision Models](https://mlsanity.com/paper/2605.13675/) (2026)
- [Geometry matters: insights from Ollivier Ricci Curvature and Ricci Flow into representational alignment through Ollivier-Ricci Curvature and Ricci Flow](https://mlsanity.com/paper/2501.00919/) (2025)
- [Shifting Attention to You: Personalized Brain-Inspired AI Models](https://mlsanity.com/paper/2502.04658/) (2025)
- [Learning Fourier shapes to probe the geometric world of deep neural networks](https://mlsanity.com/paper/2511.04970/) (2025)
- [Fine-Grained Generalization via Structuralizing Concept and Feature Space into Commonality, Specificity and Confounding](https://mlsanity.com/paper/2601.03056/) (2026)
- [Context Sensitivity Improves Human-Machine Visual Alignment](https://mlsanity.com/paper/2604.13883/) (2026)
- [Do Large Language Models know who did what to whom?](https://mlsanity.com/paper/2504.16884/) (2025)
- [Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment](https://mlsanity.com/paper/2505.16419/) (2025)
- [Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context](https://mlsanity.com/paper/2507.12889/) (2025)
- [Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions](https://mlsanity.com/paper/2507.14549/) (2025)
- [Experience Scaling: Post-Deployment Evolution For Large Language Models](https://mlsanity.com/paper/2509.18771/) (2025)
- [Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports](https://mlsanity.com/paper/2509.24298/) (2025)
- [Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception](https://mlsanity.com/paper/2510.01502/) (2025)
- [Temporal Slowness in Central Vision Drives Semantic Object Learning](https://mlsanity.com/paper/2602.04462/) (2026)
- [Similarity-based representation factorization for revealing interpretable dimensions in representational data](https://mlsanity.com/paper/2605.26921/) (2026)

## Abstract (arXiv)

Determining the similarities and differences between humans and artificial intelligence (AI) is an important goal both in computational cognitive neuroscience and machine learning, promising a deeper understanding of human cognition and safer, more reliable AI systems. Much previous work comparing representations in humans and AI has relied on global, scalar measures to quantify their alignment. However, without explicit hypotheses, these measures only inform us about the degree of alignment, not the factors that determine it. To address this challenge, we propose a generic framework to compare human and AI representations, based on identifying latent representational dimensions underlying the same behavior in both domains. Applying this framework to humans and a deep neural network (DNN) model of natural images revealed a low-dimensional DNN embedding of both visual and semantic dimensions. In contrast to humans, DNNs exhibited a clear dominance of visual over semantic properties, indicating divergent strategies for representing images. While in-silico experiments showed seemingly consistent interpretability of DNN dimensions, a direct comparison between human and DNN representations revealed substantial differences in how they process images. By making representations directly comparable, our results reveal important challenges for representational alignment and offer a means for improving their comparability.