# Improving neural network representations using human similarity judgments

arXiv: https://arxiv.org/abs/2306.04507 · HTML: https://mlsanity.com/paper/2306.04507/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 94 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)

## Built on by (most cited first, 30 of 38)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Interpreting the linear structure of vision-language model embedding spaces](https://mlsanity.com/paper/2504.11695/) (2025)
- [It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap](https://mlsanity.com/paper/2405.18570/) (2024)
- [On Benchmarking Human-Like Intelligence in Machines](https://mlsanity.com/paper/2502.20502/) (2025)
- [Learned feature representations are biased by complexity, learning order, position, and more](https://mlsanity.com/paper/2405.05847/) (2024)
- [Dimensions underlying the representational alignment of deep neural networks with humans](https://mlsanity.com/paper/2406.19087/) (2024)
- [When Does Perceptual Alignment Benefit Vision Representations?](https://mlsanity.com/paper/2410.10817/) (2024)
- [A Spectral Theory of Neural Prediction and Alignment](https://mlsanity.com/paper/2309.12821/) (2023)
- [AI-based Anomaly Detection for Clinical-Grade Histopathological Diagnostics](https://mlsanity.com/paper/2406.14866/) (2024)
- [Learning Human-like Representations to Enable Learning Human Values](https://mlsanity.com/paper/2312.14106/) (2023)
- [Human-Aligned Image Models Improve Visual Decoding from the Brain](https://mlsanity.com/paper/2502.03081/) (2025)
- [An Analysis of Human Alignment of Latent Diffusion Models](https://mlsanity.com/paper/2403.08469/) (2024)
- [CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations](https://mlsanity.com/paper/2404.16482/) (2024)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Set Learning for Accurate and Calibrated Models](https://mlsanity.com/paper/2307.02245/) (2023)
- [Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI](https://mlsanity.com/paper/2401.11835/) (2024)
- [Brain-aligning of semantic vectors improves neural decoding of visual stimuli](https://mlsanity.com/paper/2403.15176/) (2024)
- [Alignment and Adversarial Robustness: Are More Human-Like Models More Secure?](https://mlsanity.com/paper/2502.12377/) (2025)
- [Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts](https://mlsanity.com/paper/2505.13281/) (2025)
- [Evaluating Steering Techniques using Human Similarity Judgments](https://mlsanity.com/paper/2505.19333/) (2025)
- [Characterizing Universal Object Representations Across Vision Models](https://mlsanity.com/paper/2605.13675/) (2026)
- [fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception in image embeddings](https://mlsanity.com/paper/2406.01278/) (2024)
- [Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks](https://mlsanity.com/paper/2409.06362/) (2024)
- [Shifting Attention to You: Personalized Brain-Inspired AI Models](https://mlsanity.com/paper/2502.04658/) (2025)
- [Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments](https://mlsanity.com/paper/2504.07965/) (2025)
- [Semantic Depth Matters: Explaining Errors of Deep Vision Networks through Perceived Class Similarities](https://mlsanity.com/paper/2504.09956/) (2025)
- [Low-Pass Filtering Improves Behavioral Alignment of Vision Models](https://mlsanity.com/paper/2602.13859/) (2026)
- [Context Sensitivity Improves Human-Machine Visual Alignment](https://mlsanity.com/paper/2604.13883/) (2026)
- [Extremely coarse learning objectives induce human-aligned representations in AI vision models](https://mlsanity.com/paper/2605.05556/) (2026)
- [Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks](https://mlsanity.com/paper/2503.02147/) (2025)

## Abstract (arXiv)

Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning it with human similarity judgments. We find that a naive approach leads to large changes in local representational structure that harm downstream performance. Thus, we propose a novel method that aligns the global structure of representations while preserving their local structure. This global-local transform considerably improves accuracy across a variety of few-shot learning and anomaly detection tasks. Our results indicate that human visual representations are globally organized in a way that facilitates learning from few examples, and incorporating this global structure into neural network representations improves performance on downstream tasks.