# Discovering Divergent Representations between Text-to-Image Models

arXiv: https://arxiv.org/abs/2509.08940 · HTML: https://mlsanity.com/paper/2509.08940/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation](https://mlsanity.com/paper/2402.17245/) (2024)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://mlsanity.com/paper/2202.04053/) (2022)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [OpenBias: Open-set Bias Detection in Text-to-Image Generative Models](https://mlsanity.com/paper/2404.07990/) (2024)
- [Describing Differences in Image Sets with Natural Language](https://mlsanity.com/paper/2312.02974/) (2023)
- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [Rosetta Neurons: Mining the Common Units in a Model Zoo](https://mlsanity.com/paper/2306.09346/) (2023)
- [T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation](https://mlsanity.com/paper/2306.00905/) (2023)
- [Mass-Producing Failures of Multimodal Systems with Language Models](https://mlsanity.com/paper/2306.12105/) (2023)
- [Interpreting the Second-Order Effects of Neurons in CLIP](https://mlsanity.com/paper/2406.04341/) (2024)
- ['Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion](https://mlsanity.com/paper/2310.19981/) (2023)
- [Identifying Implicit Social Biases in Vision-Language Models](https://mlsanity.com/paper/2411.00997/) (2024)
- [TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generation](https://mlsanity.com/paper/2307.05134/) (2023)

## Built on by (most cited first, 2 of 2)

- [DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation](https://mlsanity.com/paper/2607.00183/) (2026)
- [Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models](https://mlsanity.com/paper/2610.18990/) (2026)

## Abstract (arXiv)

In this paper, we investigate when and how visual representations learned by two different generative models diverge. Given two text-to-image models, our goal is to discover visual attributes that appear in images generated by one model but not the other, along with the types of prompts that trigger these attribute differences. For example, "flames" might appear in one model's outputs when given prompts expressing strong emotions, while the other model does not produce this attribute given the same prompts. We introduce CompCon (Comparing Concepts), an evolutionary search algorithm that discovers visual attributes more prevalent in one model's output than the other, and uncovers the prompt concepts linked to these visual differences. To evaluate CompCon's ability to find diverging representations, we create an automated data generation pipeline to produce ID2, a dataset of 60 input-dependent differences, and compare our approach to several LLM- and VLM-powered baselines. Finally, we use CompCon to compare popular text-to-image models, finding divergent representations such as how PixArt depicts prompts mentioning loneliness with wet streets and Stable Diffusion 3.5 depicts African American people in media professions. Code at: https://github.com/adobe-research/CompCon