# Words are all you need? Language as an approximation for human similarity judgments

arXiv: https://arxiv.org/abs/2206.04105 · HTML: https://mlsanity.com/paper/2206.04105/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 31 of its 68 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO](https://mlsanity.com/paper/2004.15020/) (2020)
- [PyTorchVideo: A Deep Learning Library for Video Understanding](https://mlsanity.com/paper/2111.09887/) (2021)
- [Gibbs Sampling with People](https://mlsanity.com/paper/2008.02595/) (2020)
- [Variations of the Similarity Function of TextRank for Automated Summarization](https://mlsanity.com/paper/1602.03606/) (2016)
- [Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings](https://mlsanity.com/paper/2011.11015/) (2020)
- [Extracting low-dimensional psychological representations from convolutional neural networks](https://mlsanity.com/paper/2005.14363/) (2020)
- [Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics](https://mlsanity.com/paper/1805.08501/) (2018)
- [Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity](https://mlsanity.com/paper/2107.07013/) (2021)

## Built on by (most cited first, 18 of 18)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [What Do Language Models Hear? Probing for Auditory Representations in Language Models](https://mlsanity.com/paper/2402.16998/) (2024)
- [Do Large Language Models Mirror Cognitive Language Processing?](https://mlsanity.com/paper/2402.18023/) (2024)
- [Learning Human-like Representations to Enable Learning Human Values](https://mlsanity.com/paper/2312.14106/) (2023)
- [An Analysis of Human Alignment of Latent Diffusion Models](https://mlsanity.com/paper/2403.08469/) (2024)
- [CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations](https://mlsanity.com/paper/2404.16482/) (2024)
- [The Universal Law of Generalization Holds for Naturalistic Stimuli](https://mlsanity.com/paper/2306.08564/) (2023)
- [TurtleBench: A Visual Programming Benchmark in Turtle Geometry](https://mlsanity.com/paper/2411.00264/) (2024)
- [Dimensions of Disagreement: Unpacking Divergence and Misalignment in Cognitive Science and Artificial Intelligence](https://mlsanity.com/paper/2310.12994/) (2023)
- [Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People](https://mlsanity.com/paper/2406.04278/) (2024)
- [Analyzing the Roles of Language and Vision in Learning from Limited Data](https://mlsanity.com/paper/2403.19669/) (2024)
- [A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysis](https://mlsanity.com/paper/2412.00577/) (2024)
- [Are Expressions for Music Emotions the Same Across Cultures?](https://mlsanity.com/paper/2502.08744/) (2025)
- [Reason to Play: Behavioral and Brain Alignment Between Frontier LRMs and Human Game Learners](https://mlsanity.com/paper/2605.08019/) (2026)
- [Towards Precision Characterization of Communication Disorders using Models of Perceived Pragmatic Similarity](https://mlsanity.com/paper/2409.09170/) (2024)
- [Translation from the Information Bottleneck Perspective: an Efficiency Analysis of Spatial Prepositions in Bitexts](https://mlsanity.com/paper/2603.19924/) (2026)
- [What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?](https://mlsanity.com/paper/2607.16214/) (2026)

## Abstract (arXiv)

Human similarity judgments are a powerful supervision signal for machine learning applications based on techniques such as contrastive learning, information retrieval, and model alignment, but classical methods for collecting human similarity judgments are too expensive to be used at scale. Recent methods propose using pre-trained deep neural networks (DNNs) to approximate human similarity, but pre-trained DNNs may not be available for certain domains (e.g., medical images, low-resource languages) and their performance in approximating human similarity has not been extensively tested. We conducted an evaluation of 611 pre-trained models across three domains -- images, audio, video -- and found that there is a large gap in performance between human similarity judgments and pre-trained DNNs. To address this gap, we propose a new class of similarity approximation methods based on language. To collect the language data required by these new methods, we also developed and validated a novel adaptive tag collection pipeline. We find that our proposed language-based methods are significantly cheaper, in the number of human judgments, than classical methods, but still improve performance over the DNN-based methods. Finally, we also develop `stacked' methods that combine language embeddings with DNN embeddings, and find that these consistently provide the best approximations for human similarity across all three of our modalities. Based on the results of this comprehensive study, we provide a concise guide for researchers interested in collecting or approximating human similarity data. To accompany this guide, we also release all of the similarity and language data, a total of 206,339 human judgments, that we collected in our experiments, along with a detailed breakdown of all modeling results.