mlsanityData to 14 Oct 2026

Improving neural network representations using human similarity judgments

arXiv 2306.04507 ↗First posted 7 Jun 2023cs.CV cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning it with human similarity judgments. We find that a naive approach leads to large changes in local representational structure that harm downstream performance. Thus, we propose a novel method that aligns the global structure of representations while preserving their local structure. This global-local transform considerably improves accuracy across a variety of few-shot learning and anomaly detection tasks. Our results indicate that human visual representations are globally organized in a way that facilitates learning from few examples, and incorporating this global structure into neural network representations improves performance on downstream tasks.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 52 of its 94 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

52 of 94 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2017Prototypical Networks for Few-shot Learningcited by 3,015
38 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2019Invariant Risk Minimizationcited by 1,854
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  8. 2018Recognition in Terra Incognitacited by 658
  9. 2018Do Better ImageNet Models Transfer Better?cited by 565
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  11. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  12. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  13. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 365
  14. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  15. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  16. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  17. 2018Generalisation in humans and deep neural networkscited by 237
  18. 2019Human uncertainty makes classification more robustcited by 235
  19. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  20. 2016What makes ImageNet good for transfer learning?cited by 192
  21. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  22. 2021Partial success in closing the gap between human and machine visioncited by 185
  23. 2020PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentationcited by 175
  24. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  25. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 126
  26. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
  27. 2020Deep Nearest Neighbor Anomaly Detectioncited by 104
  28. 2022Human alignment of neural network representationscited by 74
  29. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  30. 2017Do Convolutional Neural Networks Learn Class Hierarchy?cited by 61
  31. 2022Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Imagescited by 35
  32. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  33. 2019Baby steps towards few-shot learning with multiple semanticscited by 29
  34. 2020On the surprising similarities between supervised and self-supervised modelscited by 29
  35. 2022From t-SNE to UMAP with contrastive learningcited by 28
  36. 2021Transformaly -- Two (Feature Spaces) Are Better Than Onecited by 25
  37. 2023Alignment with human representations supports robust few-shot learningcited by 25
  38. 2020Transforming Neural Network Visual Representations to Predict Human Judgments of Similaritycited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 12; 2025: 13; 2026: 9.

38 papers.

Sort:

By area:

  1. 2023Getting aligned on representational alignmentcited by 143
  2. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  3. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  4. 2025On Benchmarking Human-Like Intelligence in Machinescited by 25
  5. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 23
  6. 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
  7. 2024When Does Perceptual Alignment Benefit Vision Representations?cited by 18
  8. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
  9. 2024AI-based Anomaly Detection for Clinical-Grade Histopathological Diagnosticscited by 8
  10. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  11. 2025Human-Aligned Image Models Improve Visual Decoding from the Braincited by 7
  12. 2024An Analysis of Human Alignment of Latent Diffusion Modelscited by 5
26 more
  1. 2024CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representationscited by 5
  2. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  3. 2023Set Learning for Accurate and Calibrated Modelscited by 2
  4. 2024Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AIcited by 2
  5. 2024Brain-aligning of semantic vectors improves neural decoding of visual stimulicited by 2
  6. 2025Alignment and Adversarial Robustness: Are More Human-Like Models More Secure?cited by 2
  7. 2025Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Conceptscited by 2
  8. 2025Evaluating Steering Techniques using Human Similarity Judgmentscited by 2
  9. 2026Characterizing Universal Object Representations Across Vision Modelscited by 2
  10. 2024fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception in image embeddingscited by 1
  11. 2024Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networkscited by 1
  12. 2025Shifting Attention to You: Personalized Brain-Inspired AI Modelscited by 1
  13. 2025Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgmentscited by 1
  14. 2025Semantic Depth Matters: Explaining Errors of Deep Vision Networks through Perceived Class Similaritiescited by 1
  15. 2026Low-Pass Filtering Improves Behavioral Alignment of Vision Modelscited by 1
  16. 2026Context Sensitivity Improves Human-Machine Visual Alignmentcited by 1
  17. 2026Extremely coarse learning objectives induce human-aligned representations in AI vision modelscited by 1
  18. 2025Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Trickscited by 0
  19. 2025Do Large Language Models know who did what to whom?cited by 0
  20. 2025Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignmentcited by 0
  21. 2025Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perceptioncited by 0
  22. 2026Rank-based Geographical Regularization: Revisiting Contrastive Self-Supervised Learning for Multispectral Remote Sensing Imagerycited by 0
  23. 2026Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulationcited by 0
  24. 2026Capability neq Interpretability: Human Interpretability of Vision Foundation Modelscited by 0
  25. 2026AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relationscited by 0
  26. 2026Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similaritycited by 0