# Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings

arXiv: https://arxiv.org/abs/2011.11015 · HTML: https://mlsanity.com/paper/2011.11015/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2020.

## Built on by (most cited first, 23 of 23)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs](https://mlsanity.com/paper/2302.02865/) (2023)
- [Words are all you need? Language as an approximation for human similarity judgments](https://mlsanity.com/paper/2206.04105/) (2022)
- [VICE: Variational Interpretable Concept Embeddings](https://mlsanity.com/paper/2205.00756/) (2022)
- [On the Informativeness of Supervision Signals](https://mlsanity.com/paper/2211.01407/) (2022)
- [Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception](https://mlsanity.com/paper/2306.03229/) (2023)
- [A View From Somewhere: Human-Centric Face Representations](https://mlsanity.com/paper/2303.17176/) (2023)
- [CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations](https://mlsanity.com/paper/2404.16482/) (2024)
- [Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergence](https://mlsanity.com/paper/2409.09413/) (2024)
- [The Universal Law of Generalization Holds for Naturalistic Stimuli](https://mlsanity.com/paper/2306.08564/) (2023)
- [Do Invariances in Deep Neural Networks Align with Human Perception?](https://mlsanity.com/paper/2111.14726/) (2021)
- [From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs](https://mlsanity.com/paper/2602.00628/) (2026)
- [Grounding Psychological Shape Space in Convolutional Neural Networks](https://mlsanity.com/paper/2111.08409/) (2021)
- [Visual Knowledge Tracing](https://mlsanity.com/paper/2207.10157/) (2022)
- [Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization](https://mlsanity.com/paper/2503.00228/) (2025)
- [Context Sensitivity Improves Human-Machine Visual Alignment](https://mlsanity.com/paper/2604.13883/) (2026)
- [Understanding top-down attention using task-oriented ablation design](https://mlsanity.com/paper/2106.11339/) (2021)
- [The challenge of representation learning: Improved accuracy in deep vision models does not come with better predictions of perceptual similarity](https://mlsanity.com/paper/2303.07084/) (2023)
- [Probabilistic Contrastive Learning with Explicit Concentration on the Hypersphere](https://mlsanity.com/paper/2405.16460/) (2024)
- [Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment](https://mlsanity.com/paper/2505.16419/) (2025)
- [Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity](https://mlsanity.com/paper/2609.32876/) (2026)

## Abstract (arXiv)

Advances in object recognition flourished in part because of the availability of high-quality datasets and associated benchmarks. However, these benchmarks---such as ILSVRC---are relatively task-specific, focusing predominately on predicting class labels. We introduce a publicly-available dataset that embodies the task-general capabilities of human perception and reasoning. The Human Similarity Judgments extension to ImageNet (ImageNet-HSJ) is composed of human similarity judgments that supplement the ILSVRC validation set. The new dataset supports a range of task and performance metrics, including the evaluation of unsupervised learning algorithms. We demonstrate two methods of assessment: using the similarity judgments directly and using a psychological embedding trained on the similarity judgments. This embedding space contains an order of magnitude more points (i.e., images) than previous efforts based on human judgments. Scaling to the full 50,000 image set was made possible through a selective sampling process that used variational Bayesian inference and model ensembles to sample aspects of the embedding space that were most uncertain. This methodological innovation not only enables scaling, but should also improve the quality of solutions by focusing sampling where it is needed. To demonstrate the utility of ImageNet-HSJ, we used the similarity ratings and the embedding space to evaluate how well several popular models conform to human similarity judgments. One finding is that more complex models that perform better on task-specific benchmarks do not better conform to human semantic judgments. In addition to the human similarity judgments, pre-trained psychological embeddings and code for inferring variational embeddings are made publicly available. Collectively, ImageNet-HSJ assets support the appraisal of internal representations and the development of more human-like models.