# Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity

arXiv: https://arxiv.org/abs/2010.06512 · HTML: https://mlsanity.com/paper/2010.06512/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [Do better ImageNet classifiers assess perceptual similarity better?](https://mlsanity.com/paper/2203.04946/) (2022)
- [Dimensions underlying the representational alignment of deep neural networks with humans](https://mlsanity.com/paper/2406.19087/) (2024)
- [A View From Somewhere: Human-Centric Face Representations](https://mlsanity.com/paper/2303.17176/) (2023)
- [Grounding Psychological Shape Space in Convolutional Neural Networks](https://mlsanity.com/paper/2111.08409/) (2021)
- [Visual Knowledge Tracing](https://mlsanity.com/paper/2207.10157/) (2022)
- [Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks](https://mlsanity.com/paper/2409.06362/) (2024)
- [Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization](https://mlsanity.com/paper/2503.00228/) (2025)
- [Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments](https://mlsanity.com/paper/2504.07965/) (2025)
- [Context Sensitivity Improves Human-Machine Visual Alignment](https://mlsanity.com/paper/2604.13883/) (2026)
- [Explaining Human Comparisons using Alignment-Importance Heatmaps](https://mlsanity.com/paper/2409.16292/) (2024)
- [CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing](https://mlsanity.com/paper/2511.08512/) (2025)
- [Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision](https://mlsanity.com/paper/2604.21909/) (2026)
- [Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision](https://mlsanity.com/paper/2608.27877/) (2026)

## Abstract (arXiv)

Deep-learning vision models have shown intriguing similarities and differences with respect to human vision. We investigate how to bring machine visual representations into better alignment with human representations. Human representations are often inferred from behavioral evidence such as the selection of an image most similar to a query image. We find that with appropriate linear transformations of deep embeddings, we can improve prediction of human binary choice on a data set of bird images from 72% at baseline to 89%. We hypothesized that deep embeddings have redundant, high (4096) dimensional representations; however, reducing the rank of these representations results in a loss of explanatory power. We hypothesized that the dilation transformation of representations explored in past research is too restrictive, and indeed we found that model explanatory power can be significantly improved with a more expressive linear transform. Most surprising and exciting, we found that, consistent with classic psychological literature, human similarity judgments are asymmetric: the similarity of X to Y is not necessarily equal to the similarity of Y to X, and allowing models to express this asymmetry improves explanatory power.