# Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding

arXiv: https://arxiv.org/abs/2402.08919 · HTML: https://mlsanity.com/paper/2402.08919/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://mlsanity.com/paper/2202.08904/) (2022)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [PromptBERT: Improving BERT Sentence Embeddings with Prompts](https://mlsanity.com/paper/2201.04337/) (2022)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO](https://mlsanity.com/paper/2004.15020/) (2020)
- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [Towards Conceptual Compression](https://mlsanity.com/paper/1604.08772/) (2016)
- [Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models](https://mlsanity.com/paper/2306.05357/) (2023)
- [FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations](https://mlsanity.com/paper/2203.16639/) (2022)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Semantic bottleneck for computer vision tasks](https://mlsanity.com/paper/1811.02234/) (2018)

## Built on by (most cited first, 4 of 4)

- [AI Agents as Universal Task Solvers](https://mlsanity.com/paper/2510.12066/) (2025)
- [ConViS-Bench: Estimating Video Similarity Through Semantic Concepts](https://mlsanity.com/paper/2509.19245/) (2025)
- [Prompting Complexity: Shortest Prompts for Texts and Behaviors in LLMs](https://mlsanity.com/paper/2607.06145/) (2026)
- [Conjuring Semantic Similarity](https://mlsanity.com/paper/2410.16431/) (2024)

## Abstract (arXiv)

Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning. In legal doctrine however, determining the degree of similarity between works requires subjective analysis, and fact-finders (judges and juries) can demonstrate considerable variability in these subjective judgement calls. Images that are structurally similar can be deemed dissimilar, whereas images of completely different scenes can be deemed similar enough to support a claim of copying. We seek to define and compute a notion of "conceptual similarity" among images that captures high-level relations even among images that do not share repeated elements or visually similar components. The idea is to use a base multi-modal model to generate "explanations" (captions) of visual data at increasing levels of complexity. Then, similarity can be measured by the length of the caption needed to discriminate between the two images: Two highly dissimilar images can be discriminated early in their description, whereas conceptually dissimilar ones will need more detail to be distinguished. We operationalize this definition and show that it correlates with subjective (averaged human evaluation) assessment, and beats existing baselines on both image-to-image and text-to-text similarity benchmarks. Beyond just providing a number, our method also offers interpretability by pointing to the specific level of granularity of the description where the source data are differentiated.