# Totally Looks Like - How Humans Compare, Compared to Machines

arXiv: https://arxiv.org/abs/1803.01485 · HTML: https://mlsanity.com/paper/1803.01485/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2018.

## Built on by (most cited first, 12 of 12)

- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming](https://mlsanity.com/paper/2505.16667/) (2025)
- [Learning to Complement and to Defer to Multiple Users](https://mlsanity.com/paper/2407.07003/) (2024)
- [Learning to Complement with Multiple Humans](https://mlsanity.com/paper/2311.13172/) (2023)
- [Seeing Faces in Things: A Model and Dataset for Pareidolia](https://mlsanity.com/paper/2409.16143/) (2024)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [Vibe Spaces for Creatively Connecting and Expressing Visual Concepts](https://mlsanity.com/paper/2512.14884/) (2025)
- [Learning an Adaptation Function to Assess Image Visual Similarities](https://mlsanity.com/paper/2206.01417/) (2022)
- [ChannelExplorer: Exploring Class Separability Through Activation Channel Visualization](https://mlsanity.com/paper/2505.04647/) (2025)
- [Analyze-Prompt-Reason: A Collaborative Agent-Based Framework for Multi-Image Vision-Language Reasoning](https://mlsanity.com/paper/2508.00356/) (2025)
- [Relational Visual Similarity](https://mlsanity.com/paper/2512.07833/) (2025)

## Abstract (arXiv)

Perceptual judgment of image similarity by humans relies on rich internal representations ranging from low-level features to high-level concepts, scene properties and even cultural associations. However, existing methods and datasets attempting to explain perceived similarity use stimuli which arguably do not cover the full breadth of factors that affect human similarity judgments, even those geared toward this goal. We introduce a new dataset dubbed Totally-Looks-Like (TLL) after a popular entertainment website, which contains images paired by humans as being visually similar. The dataset contains 6016 image-pairs from the wild, shedding light upon a rich and diverse set of criteria employed by human beings. We conduct experiments to try to reproduce the pairings via features extracted from state-of-the-art deep convolutional neural networks, as well as additional human experiments to verify the consistency of the collected data. Though we create conditions to artificially make the matching task increasingly easier, we show that machine-extracted representations perform very poorly in terms of reproducing the matching selected by humans. We discuss and analyze these results, suggesting future directions for improvement of learned image representations.