mlsanityData to 7 Oct 2026

Harmonizing the object recognition strategies of deep neural networks with humans

arXiv 2211.04533 ↗First posted 8 Nov 2022cs.CV cs.AIcited by 59

Not on the map. Why?

Abstract (arXiv)

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition. State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves. We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns DNN and human visual strategies and improves categorization accuracy. Our work represents the first demonstration that the scaling laws that are guiding the design of DNNs today have also produced worse models of human vision. We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 65 of its 102 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

65 of 102 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2019Searching for MobileNetV3cited by 1,984
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2021Scaling Vision Transformerscited by 993
  13. 2019When Does Label Smoothing Help?cited by 985
  14. 2020Designing Network Design Spacescited by 954
  15. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  17. 2020Quantifying Attention Flow in Transformerscited by 759
  18. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  19. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  20. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  21. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 468
  22. 2020ResNeSt: Split-Attention Networkscited by 445
  23. 2021Transformer Trackingcited by 412
  24. 2019CSPNet: A New Backbone that can Enhance Learning Capability of CNNcited by 381
  25. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  26. 2019Adversarial Examples Improve Image Recognitioncited by 283
  27. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  28. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  29. 2021Partial success in closing the gap between human and machine visioncited by 185
  30. 2017Dual Path Networkscited by 119
  31. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  32. 2019Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNscited by 90
  33. 2022Human alignment of neural network representationscited by 74
  34. 2019XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Cameracited by 68
  35. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  36. 2017Superhuman Accuracy on the SNEMI3D Connectomics Challengecited by 64
  37. 2019Recurrence is required to capture the representational dynamics of the human visual systemcited by 58
  38. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 20; 2025: 10; 2026: 10.

59 papers.

Sort:

By area:

  1. 2023Getting aligned on representational alignmentcited by 142
  2. 2023Neither hype nor gloom do DNNs justicecited by 40
  3. 2023Alignment with human representations supports robust few-shot learningcited by 25
  4. 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
  5. 2023Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortexcited by 18
  6. 2024When Does Perceptual Alignment Benefit Vision Representations?cited by 18
  7. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  8. 2023Studying How to Efficiently and Effectively Guide Models with Explanationscited by 16
  9. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  10. 2023Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attentioncited by 15
  11. 2023SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstractioncited by 14
  12. 2022Spuriosity Rankings: Sorting Data to Measure and Mitigate Biasescited by 13
38 more
  1. 2023Diffusion Models as Artists: Are we Closing the Gap between Humans and Machines?cited by 10
  2. 2025Better artificial intelligence does not mean better models of biologycited by 10
  3. 2023Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perceptioncited by 7
  4. 2024A comparison between humans and AI at recognizing objects in unusual posescited by 7
  5. 2024Structured Gradient-based Interpretations via Norm-Regularized Adversarial Trainingcited by 7
  6. 2025Human-Aligned Image Models Improve Visual Decoding from the Braincited by 7
  7. 2022Using Human Perception to Regularize Transfer Learningcited by 6
  8. 2023Computing a human-like reaction time metric from stable recurrent vision modelscited by 6
  9. 2024How Aligned are Different Alignment Metrics?cited by 6
  10. 2025Representational Similarity via Interpretable Visual Conceptscited by 5
  11. 2025CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Modelscited by 5
  12. 2023Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproductscited by 4
  13. 2023Divergences in Color Perception between Deep Neural Networks and Humanscited by 4
  14. 2026Bi-Orthogonal Factor Decomposition for Vision Transformerscited by 4
  15. 2024Taking Training Seriously: Human Guidance and Management-Based Regulation of Artificial Intelligencecited by 3
  16. 2024Representational Alignment Supports Effective Machine Teachingcited by 3
  17. 2024Tracking objects that change in appearance with phase synchronycited by 3
  18. 2024RTify: Aligning Deep Neural Networks with Human Behavioral Decisionscited by 3
  19. 2023Spider GAN: Leveraging Friendly Neighbors to Accelerate GAN Trainingcited by 2
  20. 2024The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approachcited by 2
  21. 2024L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancementcited by 2
  22. 2025AIM: Amending Inherent Interpretability via Self-Supervised Maskingcited by 2
  23. 2023Beyond the limitations of any imaginable mechanism: large language models and psycholinguisticscited by 1
  24. 2023Layerwise complexity-matched learning yields an improved model of cortical area V2cited by 1
  25. 2024VFA: Vision Frequency Analysis of Foundation Models and Humancited by 1
  26. 2024Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networkscited by 1
  27. 2024Artificial Human Intelligence: The role of Humans in the Development of Next Generation AIcited by 1
  28. 2024Increasing Interpretability of Neural Networks By Approximating Human Visual Saliencycited by 1
  29. 2025Enhancing Vision Transformer Explainability Using Artificial Astrocytescited by 1
  30. 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMscited by 1
  31. 2026Extremely coarse learning objectives induce human-aligned representations in AI vision modelscited by 1
  32. 2022Can Machines Imitate Humans? Integrative Turing-like tests for Language and Vision Demonstrate a Narrowing Gapcited by 0
  33. 2023Fixing the problems of deep neural networks will require better training data and learning algorithmscited by 0
  34. 2024Graphical Perception of Saliency-based Model Explanationscited by 0
  35. 2024Dilated Convolution with Learnable Spacings makes visual models more aligned with humans: a Grad-CAM studycited by 0
  36. 2024Training Better Deep Learning Models Using Human Saliencycited by 0
  37. 2025Quantifying task-relevant representational similarity using decision variable correlationcited by 0
  38. 2025Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariancecited by 0

50 most cited of 59 All 59 (JSON)