mlsanityData to 14 Oct 2026

Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

arXiv 2407.12854 ↗First posted 9 Jul 2024cs.CL cs.AI cs.IRcited by 56

Not on the map. Why?

Cited by 22 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in different configurations. In this paper, we consider another dimension of scaling: the amount of data available at inference time. Specifically, we find that increasing the size of the datastore used by a retrieval-based LM monotonically improves language modeling and several downstream tasks without obvious saturation, such that a smaller model augmented with a large datastore outperforms a larger LM-only model on knowledge-intensive tasks. By plotting compute-optimal scaling curves with varied datastore, model, and pretraining data sizes, we show that using larger datastores can significantly improve model performance for the same training compute budget. We carry out our study by constructing a 1.4 trillion-token datastore named MassiveDS, which is the largest and the most diverse open-sourced datastore for retrieval-based LMs to date, and designing an efficient pipeline for studying datastore scaling in a computationally accessible manner. Finally, we analyze the effect of improving the retriever, datastore quality filtering, and other design choices on our observed scaling trends. Overall, our results show that datastore size should be considered as an integral part of LM efficiency and performance trade-offs. To facilitate future research, we open-source our datastore and code at https://github.com/RulinShao/retrieval-scaling.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

49 of 60 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
37 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  13. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  14. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  15. 2019Passage Re-ranking with BERTcited by 645
  16. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  17. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  18. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  19. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  20. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  21. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  22. 2023Scaling Data-Constrained Language Modelscited by 403
  23. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  24. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  25. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  26. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  27. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 165
  28. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  29. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  30. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
  31. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  32. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  33. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54
  34. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  35. 2022Nonparametric Masked Language Modelingcited by 40
  36. 2022Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Modelcited by 32
  37. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 33; 2026: 14.

56 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  2. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 77
  3. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  4. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  5. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 65
  6. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 58
  7. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  8. 2025Diffusion Language Models are Super Data Learnerscited by 48
  9. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defectscited by 27
  10. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  11. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  12. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
38 more
  1. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  2. 2025Agentic Learner with Grow-and-Refine Multimodal Semantic Memorycited by 13
  3. 2025VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correctioncited by 12
  4. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  5. 2024Improving Factuality with Explicit Working Memorycited by 10
  6. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  7. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  8. 2024Towards flexible perception with visual memorycited by 8
  9. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reusecited by 8
  10. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarkscited by 8
  11. 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7
  12. 2025FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factualitycited by 6
  13. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 5
  14. 2025OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learningcited by 4
  15. 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaborationcited by 4
  16. 2025ConfRAG: Confidence-Guided Retrieval-Augmenting Generationcited by 4
  17. 2025LEANN: A Low-Storage Vector Indexcited by 4
  18. 2025Provable Benefits of In-Tool Learning for Large Language Modelscited by 4
  19. 2025Less LLM, More Documents: Searching for Improved RAGcited by 4
  20. 2025Reusing Pre-Training Data at Test Time is a Compute Multipliercited by 4
  21. 2024What External Knowledge is Preferred by LLMs? Characterizing and Exploring Chain of Evidence in Imperfect Context for Multi-Hop QAcited by 3
  22. 2025Query-driven Document-level Scientific Evidence Extraction from Biomedical Studiescited by 3
  23. 2025RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentationcited by 3
  24. 2025Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performancecited by 2
  25. 2026How Retrieved Context Shapes Internal Representations in RAGcited by 2
  26. 2025DS SERVE: A Framework for Efficient and Scalable Neural Retrievalcited by 2
  27. 2026RAG over Thinking Traces Can Improve Reasoning Taskscited by 2
  28. 2026PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generationcited by 2
  29. 2025Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifoldscited by 1
  30. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrievalcited by 1
  31. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  32. 2025The SMeL Test: A simple benchmark for media literacy in language modelscited by 0
  33. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0
  34. 2025What's the next frontier for Data-centric AI? Data Savvy Agentscited by 0
  35. 2025The Road of Adaptive AI for Precision in Cybersecuritycited by 0
  36. 2026Learning to Detect Language Model Training Data via Active Reconstructioncited by 0
  37. 2026Content Creation with Spillovers: An Incentive Design Approachcited by 0
  38. 2026Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?cited by 0

50 most cited of 56 All 56 (JSON)