# Scaling Laws For Dense Retrieval

arXiv: https://arxiv.org/abs/2403.18684 · HTML: https://mlsanity.com/paper/2403.18684/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 63 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://mlsanity.com/paper/2112.07577/) (2021)

## Built on by (most cited first, 29 of 29)

- [Skill Retrieval Augmentation for Agentic AI](https://mlsanity.com/paper/2604.24594/) (2026)
- [Parametric Retrieval Augmented Generation](https://mlsanity.com/paper/2501.15915/) (2025)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [LeKUBE: A Legal Knowledge Update BEnchmark](https://mlsanity.com/paper/2407.14192/) (2024)
- [Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling](https://mlsanity.com/paper/2510.03199/) (2025)
- [Drowning in Documents: Consequences of Scaling Reranker Inference](https://mlsanity.com/paper/2411.11767/) (2024)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [Exploring the Design Space of Visual Context Representation in Video MLLMs](https://mlsanity.com/paper/2410.13694/) (2024)
- [Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models](https://mlsanity.com/paper/2506.10972/) (2025)
- [Understanding Generative Recommendation with Semantic IDs from a Model-scaling View](https://mlsanity.com/paper/2509.25522/) (2025)
- [Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions](https://mlsanity.com/paper/2503.00501/) (2025)
- [Exploring Training and Inference Scaling Laws in Generative Retrieval](https://mlsanity.com/paper/2503.18941/) (2025)
- [Scaling Laws for Online Advertisement Retrieval](https://mlsanity.com/paper/2411.13322/) (2024)
- [Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs](https://mlsanity.com/paper/2508.17400/) (2025)
- [Scaling Laws for Embedding Dimension in Information Retrieval](https://mlsanity.com/paper/2602.05062/) (2026)
- [Foundations of GenIR](https://mlsanity.com/paper/2501.02842/) (2025)
- [Relative-Based Scaling Law for Neural Language Models](https://mlsanity.com/paper/2510.20387/) (2025)
- [Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning](https://mlsanity.com/paper/2603.01471/) (2026)
- [Information Anxiety in Large Language Models](https://mlsanity.com/paper/2411.10813/) (2024)
- [Which Programming Language and Model Work Best With LLM-as-a-Judge For Code Retrieval?](https://mlsanity.com/paper/2510.00324/) (2025)
- [Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.17354/) (2025)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles](https://mlsanity.com/paper/2605.01407/) (2026)
- [Compact Geometric Representations of Hierarchies](https://mlsanity.com/paper/2606.18520/) (2026)
- [Population-Level Generative Modeling for Ranking Data](https://mlsanity.com/paper/2608.08422/) (2026)
- [Web Scale on a Budget: Tail-Aware Retrieval Evaluation](https://mlsanity.com/paper/2610.17986/) (2026)

## Abstract (arXiv)

Scaling up neural models has yielded significant advancements in a wide array of tasks, particularly in language generation. Previous studies have found that the performance of neural models frequently adheres to predictable scaling laws, correlated with factors such as training set size and model size. This insight is invaluable, especially as large-scale experiments grow increasingly resource-intensive. Yet, such scaling law has not been fully explored in dense retrieval due to the discrete nature of retrieval metrics and complex relationships between training data and model sizes in retrieval tasks. In this study, we investigate whether the performance of dense retrieval models follows the scaling law as other neural models. We propose to use contrastive log-likelihood as the evaluation metric and conduct extensive experiments with dense retrieval models implemented with different numbers of parameters and trained with different amounts of annotated data. Results indicate that, under our settings, the performance of dense retrieval models follows a precise power-law scaling related to the model size and the number of annotations. Additionally, we examine scaling with prevalent data augmentation methods to assess the impact of annotation quality, and apply the scaling law to find the best resource allocation strategy under a budget constraint. We believe that these insights will significantly contribute to understanding the scaling effect of dense retrieval models and offer meaningful guidance for future research endeavors.