# Simple Entity-Centric Questions Challenge Dense Retrievers

arXiv: https://arxiv.org/abs/2109.08535 · HTML: https://mlsanity.com/paper/2109.08535/

## Key facts

- Cited by 140 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [Learning Dense Representations of Phrases at Scale](https://mlsanity.com/paper/2012.12624/) (2020)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [Efficient One-Pass End-to-End Entity Linking for Questions](https://mlsanity.com/paper/2010.02413/) (2020)
- [Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations](https://mlsanity.com/paper/2102.10073/) (2021)
- [Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP](https://mlsanity.com/paper/2106.06830/) (2021)

## Built on by (most cited first, 30 of 140)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [Dense X Retrieval: What Retrieval Granularity Should We Use?](https://mlsanity.com/paper/2312.06648/) (2023)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Automatic Evaluation of Attribution by Large Language Models](https://mlsanity.com/paper/2305.06311/) (2023)
- [Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?](https://mlsanity.com/paper/2110.06918/) (2021)
- [Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge](https://mlsanity.com/paper/2403.01432/) (2024)
- [Questions Are All You Need to Train a Dense Passage Retriever](https://mlsanity.com/paper/2206.10658/) (2022)
- [A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks](https://mlsanity.com/paper/2407.12994/) (2024)
- [Learning to Retrieve Passages without Supervision](https://mlsanity.com/paper/2112.07708/) (2021)
- [How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?](https://mlsanity.com/paper/2404.03302/) (2024)
- [Mention Memory: incorporating textual knowledge into Transformers through entity mention attention](https://mlsanity.com/paper/2110.06176/) (2021)
- [QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs](https://mlsanity.com/paper/2205.12665/) (2022)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [Rethinking the Role of Token Retrieval in Multi-Vector Retrieval](https://mlsanity.com/paper/2304.01982/) (2023)
- [WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus](https://mlsanity.com/paper/2304.04358/) (2023)
- [What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary](https://mlsanity.com/paper/2212.10380/) (2022)
- [Nearest Neighbor Speculative Decoding for LLM Generation and Attribution](https://mlsanity.com/paper/2405.19325/) (2024)
- [Decomposing Complex Queries for Tip-of-the-tongue Retrieval](https://mlsanity.com/paper/2305.15053/) (2023)
- [CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos](https://mlsanity.com/paper/2204.00716/) (2022)
- [Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs](https://mlsanity.com/paper/2408.10646/) (2024)
- [Improving Biomedical Information Retrieval with Neural Retrievers](https://mlsanity.com/paper/2201.07745/) (2022)
- [Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval](https://mlsanity.com/paper/2306.13421/) (2023)
- [Improving Code Localization with Repository Memory](https://mlsanity.com/paper/2510.01003/) (2025)

## Abstract (arXiv)

Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples. However, in this paper, we demonstrate current dense models are not yet the holy grail of retrieval. We first construct EntityQuestions, a set of simple, entity-rich questions based on facts from Wikidata (e.g., "Where was Arve Furset born?"), and observe that dense retrievers drastically underperform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions towards addressing this critical problem. First, we demonstrate that data augmentation is unable to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using specialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.