# The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes

arXiv: https://arxiv.org/abs/2012.14210 · HTML: https://mlsanity.com/paper/2012.14210/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2020.

## Built on by (most cited first, 30 of 48)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [InPars: Data Augmentation for Information Retrieval using Large Language Models](https://mlsanity.com/paper/2202.05144/) (2022)
- [Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?](https://mlsanity.com/paper/2110.06918/) (2021)
- [ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval](https://mlsanity.com/paper/2205.09153/) (2022)
- [Scaling Laws For Dense Retrieval](https://mlsanity.com/paper/2403.18684/) (2024)
- [Disentangling Dense Embeddings with Sparse Autoencoders](https://mlsanity.com/paper/2408.00657/) (2024)
- [Multi-Stage Prompting for Knowledgeable Dialogue Generation](https://mlsanity.com/paper/2203.08745/) (2022)
- [DS-TOD: Efficient Domain Specialization for Task Oriented Dialog](https://mlsanity.com/paper/2110.08395/) (2021)
- [A Thorough Examination on Zero-shot Dense Retrieval](https://mlsanity.com/paper/2204.12755/) (2022)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)
- [Modeling Exemplification in Long-form Question Answering via Retrieval](https://mlsanity.com/paper/2205.09278/) (2022)
- [Context Generation Improves Open Domain Question Answering](https://mlsanity.com/paper/2210.06349/) (2022)
- [Learning Human-like Representations to Enable Learning Human Values](https://mlsanity.com/paper/2312.14106/) (2023)
- [LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval](https://mlsanity.com/paper/2212.03222/) (2022)
- [Towards Unlocking Insights from Logbooks Using AI](https://mlsanity.com/paper/2406.12881/) (2024)
- [From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures](https://mlsanity.com/paper/2511.22150/) (2025)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models](https://mlsanity.com/paper/2312.02429/) (2023)
- [What Limits Does Quantization Place on Dense Top-k Retrieval? A Theoretical Study](https://mlsanity.com/paper/2606.11780/) (2026)
- [Large-scale Evaluation of Transformer-based Article Encoders on the Task of Citation Recommendation](https://mlsanity.com/paper/2209.05452/) (2022)
- [ProgRAG: Hallucination-Resistant Progressive Retrieval and Reasoning over Knowledge Graphs](https://mlsanity.com/paper/2511.10240/) (2025)
- [DocSCAN: Unsupervised Text Classification via Learning from Neighbors](https://mlsanity.com/paper/2105.04024/) (2021)
- [Large Language Model as a Teacher for Zero-shot Tagging at Extreme Scales](https://mlsanity.com/paper/2406.09288/) (2024)
- [Tiny Transformers Excel at Sentence Compression](https://mlsanity.com/paper/2410.23510/) (2024)
- [Fine-grained Claim-level RAG Benchmark for Law](https://mlsanity.com/paper/2605.21071/) (2026)
- [Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers](https://mlsanity.com/paper/2204.09481/) (2022)

## Abstract (arXiv)

Information Retrieval using dense low-dimensional representations recently became popular and showed out-performance to traditional sparse-representations like BM25. However, no previous work investigated how dense representations perform with large index sizes. We show theoretically and empirically that the performance for dense representations decreases quicker than sparse representations for increasing index sizes. In extreme cases, this can even lead to a tipping point where at a certain index size sparse representations outperform dense representations. We show that this behavior is tightly connected to the number of dimensions of the representations: The lower the dimension, the higher the chance for false positives, i.e. returning irrelevant documents.