# Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence

arXiv: https://arxiv.org/abs/2503.05037 · HTML: https://mlsanity.com/paper/2503.05037/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval](https://mlsanity.com/paper/2302.07452/) (2023)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [Scaling Retrieval-Based Language Models with a Trillion-Token Datastore](https://mlsanity.com/paper/2407.12854/) (2024)
- [GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers](https://mlsanity.com/paper/2205.03286/) (2022)
- [Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model](https://mlsanity.com/paper/2212.09146/) (2022)
- [DecompX: Explaining Transformers Decisions by Propagating Token Decomposition](https://mlsanity.com/paper/2306.02873/) (2023)
- [What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary](https://mlsanity.com/paper/2212.10380/) (2022)
- [MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models](https://mlsanity.com/paper/2005.02507/) (2020)
- [ABNIRML: Analyzing the Behavior of Neural IR Models](https://mlsanity.com/paper/2011.00696/) (2020)
- [COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning](https://mlsanity.com/paper/2210.15212/) (2022)
- [BERT Rankers are Brittle: a Study using Adversarial Document Perturbations](https://mlsanity.com/paper/2206.11724/) (2022)
- [Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers](https://mlsanity.com/paper/2402.13532/) (2024)
- [Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval](https://mlsanity.com/paper/2404.04163/) (2024)
- [Dense Passage Retrieval: Is it Retrieving?](https://mlsanity.com/paper/2402.11035/) (2024)
- [Boosting Big Brother: Attacking Search Engines with Encodings](https://mlsanity.com/paper/2304.14031/) (2023)

## Built on by (most cited first, 16 of 16)

- [Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrieval](https://mlsanity.com/paper/2508.19758/) (2025)
- [Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG](https://mlsanity.com/paper/2506.04390/) (2025)
- [Answer Bubbles: Information Exposure in AI-Mediated Search](https://mlsanity.com/paper/2603.16138/) (2026)
- [Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers](https://mlsanity.com/paper/2508.20408/) (2025)
- [PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark](https://mlsanity.com/paper/2601.08363/) (2026)
- [Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias](https://mlsanity.com/paper/2601.16934/) (2026)
- [ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.24388/) (2025)
- [ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge](https://mlsanity.com/paper/2506.14407/) (2025)
- [Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why](https://mlsanity.com/paper/2606.19602/) (2026)
- [Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.24977/) (2026)
- [LLM generation novelty through the lens of semantic similarity](https://mlsanity.com/paper/2510.27313/) (2025)
- [Training-Induced Bias Toward LLM-Generated Content in Dense Retrieval](https://mlsanity.com/paper/2602.10833/) (2026)
- [DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning](https://mlsanity.com/paper/2602.18633/) (2026)
- [Masking or Mitigating? Deconstructing the Impact of Query Rewriting on Retriever Biases in RAG](https://mlsanity.com/paper/2604.06097/) (2026)
- [Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors](https://mlsanity.com/paper/2606.02814/) (2026)
- [Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG](https://mlsanity.com/paper/2607.17742/) (2026)

## Abstract (arXiv)

Dense retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG). Since they often serve as the first step in these systems, their robustness is critical to avoid downstream failures. In this work, we repurpose a relation extraction dataset (e.g., Re-DocRED) to design controlled experiments that quantify the impact of heuristic biases, such as a preference for shorter documents, on retrievers like Dragon+ and Contriever. We uncover major vulnerabilities, showing retrievers favor shorter documents, early positions, repeated entities, and literal matches, all while ignoring the answer's presence! Notably, when multiple biases combine, models exhibit catastrophic performance degradation, selecting the answer-containing document in less than 10% of cases over a synthetic biased document without the answer. Furthermore, we show that these biases have direct consequences for downstream applications like RAG, where retrieval-preferred documents can mislead LLMs, resulting in a 34% performance drop than providing no documents at all. https://huggingface.co/datasets/mohsenfayyaz/ColDeR