# Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images

arXiv: https://arxiv.org/abs/2311.14084 · HTML: https://mlsanity.com/paper/2311.14084/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 23 Nov 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [AI model GPT-3 (dis)informs us better than humans](https://mlsanity.com/paper/2301.11924/) (2023)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community](https://mlsanity.com/paper/2307.09751/) (2023)
- [Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists](https://mlsanity.com/paper/1901.10437/) (2019)
- [Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media](https://mlsanity.com/paper/1704.01347/) (2017)
- [Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning](https://mlsanity.com/paper/2204.02725/) (2022)
- [A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1910.09399/) (2019)
- [Comparing Fair Ranking Metrics](https://mlsanity.com/paper/2009.01311/) (2020)
- [New Fairness Metrics for Recommendation that Embrace Differences](https://mlsanity.com/paper/1706.09838/) (2017)

## Built on by (most cited first, 13 of 13)

- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration](https://mlsanity.com/paper/2405.16546/) (2024)
- [Re-ranking the Context for Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2501.04695/) (2025)
- [AIGCs Confuse AI Too: Investigating and Explaining Synthetic Image-induced Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2403.08542/) (2024)
- [Exploring the Escalation of Source Bias in User, Data, and Recommender System Feedback Loop](https://mlsanity.com/paper/2405.17998/) (2024)
- [Retrieval Collapses When AI Pollutes the Web](https://mlsanity.com/paper/2602.16136/) (2026)
- [CalibCLIP: Contextual Calibration of Dominant Semantics for Text-Driven Image Retrieval](https://mlsanity.com/paper/2510.05586/) (2025)
- [Generative Ghost: Investigating Ranking Bias Hidden in AI-Generated Videos](https://mlsanity.com/paper/2502.07327/) (2025)
- [Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval](https://mlsanity.com/paper/2512.00804/) (2025)
- [How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews](https://mlsanity.com/paper/2604.27790/) (2026)
- [How Do LLM-Generated Texts Impact Term-Based Retrieval Models?](https://mlsanity.com/paper/2508.17715/) (2025)
- [Training-Induced Bias Toward LLM-Generated Content in Dense Retrieval](https://mlsanity.com/paper/2602.10833/) (2026)
- [Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search](https://mlsanity.com/paper/2604.08598/) (2026)

## Abstract (arXiv)

With the advancement of generation models, AI-generated content (AIGC) is becoming more realistic, flooding the Internet. A recent study suggests that this phenomenon causes source bias in text retrieval for web search. Specifically, neural retrieval models tend to rank generated texts higher than human-written texts. In this paper, we extend the study of this bias to cross-modal retrieval. Firstly, we successfully construct a suitable benchmark to explore the existence of the bias. Subsequent extensive experiments on this benchmark reveal that AI-generated images introduce an invisible relevance bias to text-image retrieval models. Specifically, our experiments show that text-image retrieval models tend to rank the AI-generated images higher than the real images, even though the AI-generated images do not exhibit more visually relevant features to the query than real images. This invisible relevance bias is prevalent across retrieval models with varying training data and architectures. Furthermore, our subsequent exploration reveals that the inclusion of AI-generated images in the training data of the retrieval models exacerbates the invisible relevance bias. The above phenomenon triggers a vicious cycle, which makes the invisible relevance bias become more and more serious. To elucidate the potential causes of invisible relevance and address the aforementioned issues, we introduce an effective training method aimed at alleviating the invisible relevance bias. Subsequently, we apply our proposed debiasing method to retroactively identify the causes of invisible relevance, revealing that the AI-generated images induce the image encoder to embed additional information into their representation. This information exhibits a certain consistency across generated images with different semantics and can make the retriever estimate a higher relevance score.