mlsanityData to 7 Oct 2026

Dimensions underlying the representational alignment of deep neural networks with humans

arXiv 2406.19087 ↗First posted 27 Jun 2024cs.CV cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Determining the similarities and differences between humans and artificial intelligence (AI) is an important goal both in computational cognitive neuroscience and machine learning, promising a deeper understanding of human cognition and safer, more reliable AI systems. Much previous work comparing representations in humans and AI has relied on global, scalar measures to quantify their alignment. However, without explicit hypotheses, these measures only inform us about the degree of alignment, not the factors that determine it. To address this challenge, we propose a generic framework to compare human and AI representations, based on identifying latent representational dimensions underlying the same behavior in both domains. Applying this framework to humans and a deep neural network (DNN) model of natural images revealed a low-dimensional DNN embedding of both visual and semantic dimensions. In contrast to humans, DNNs exhibited a clear dominance of visual over semantic properties, indicating divergent strategies for representing images. While in-silico experiments showed seemingly consistent interpretability of DNN dimensions, a direct comparison between human and DNN representations revealed substantial differences in how they process images. By making representations directly comparable, our results reveal important challenges for representational alignment and offer a means for improving their comparability.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 36 of its 87 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

36 of 87 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  10. 2018Recognition in Terra Incognitacited by 655
  11. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  12. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
24 more
  1. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  2. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 342
  3. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  4. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  5. 2018Generalisation in humans and deep neural networkscited by 237
  6. 2023Getting aligned on representational alignmentcited by 142
  7. 2018On the importance of single directions for generalizationcited by 99
  8. 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 93
  9. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  10. 2020Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Futurecited by 83
  11. 2022Human alignment of neural network representationscited by 74
  12. 2019Understanding Neural Networks via Feature Visualization: A surveycited by 65
  13. 2023On the Foundations of Shortcut Learningcited by 64
  14. 2022Harmonizing the object recognition strategies of deep neural networks with humanscited by 59
  15. 2022The neuroconnectionist research programmecited by 47
  16. 2023Improving neural network representations using human similarity judgmentscited by 38
  17. 2018Revisiting the Importance of Individual Units in CNNs via Ablationcited by 32
  18. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  19. 2019Revealing interpretable object representations from human behaviorcited by 17
  20. 2020Transforming Neural Network Visual Representations to Predict Human Judgments of Similaritycited by 15
  21. 2022VICE: Variational Interpretable Concept Embeddingscited by 15
  22. 2019Learning as the Unsupervised Alignment of Conceptual Systemscited by 14
  23. 2016Finite Sample Prediction and Recovery Bounds for Ordinal Embeddingcited by 12
  24. 2016De-Conflated Semantic Representationscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 7.

22 papers.

Sort:

By area:

  1. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  2. 2025Disentangling the Factors of Convergence between Brains and Computer Vision Modelscited by 12
  3. 2025Better artificial intelligence does not mean better models of biologycited by 10
  4. 2025Revealing emergent human-like conceptual representations from language predictioncited by 5
  5. 2025Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Eracited by 3
  6. 2026Geometric Stability: The Missing Axis of Representationscited by 2
  7. 2026Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Braincited by 2
  8. 2025Geometry matters: insights from Ollivier Ricci Curvature and Ricci Flow into representational alignment through Ollivier-Ricci Curvature and Ricci Flowcited by 1
  9. 2025Shifting Attention to You: Personalized Brain-Inspired AI Modelscited by 1
  10. 2025Learning Fourier shapes to probe the geometric world of deep neural networkscited by 1
  11. 2026Fine-Grained Generalization via Structuralizing Concept and Feature Space into Commonality, Specificity and Confoundingcited by 1
  12. 2026Context Sensitivity Improves Human-Machine Visual Alignmentcited by 1
10 more
  1. 2026Characterizing Universal Object Representations Across Vision Modelscited by 1
  2. 2025Do Large Language Models know who did what to whom?cited by 0
  3. 2025Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignmentcited by 0
  4. 2025Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental contextcited by 0
  5. 2025Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressionscited by 0
  6. 2025Experience Scaling: Post-Deployment Evolution For Large Language Modelscited by 0
  7. 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reportscited by 0
  8. 2025Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perceptioncited by 0
  9. 2026Temporal Slowness in Central Vision Drives Semantic Object Learningcited by 0
  10. 2026Similarity-based representation factorization for revealing interpretable dimensions in representational datacited by 0