mlsanityData to 14 Oct 2026

Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs

arXiv 2508.17400 ↗First posted 24 Aug 2025cs.LG cs.AI cs.IRcited by 2

Not on the map. Why?

Abstract (arXiv)

How does retrieval performance scale with pretraining FLOPs? We benchmark retrieval performance across LLM model sizes from 125 million parameters to 7 billion parameters pretrained on datasets ranging from 1 billion tokens to more than 2 trillion tokens. We find that retrieval performance on zero-shot BEIR tasks predictably scales with LLM size, training duration, and estimated FLOPs. We also show that In-Context Learning scores are strongly correlated with retrieval scores across retrieval tasks. Finally, we highlight the implications this has for the development of LLM-based retrievers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

46 of 63 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023Mistral 7Bcited by 3,164
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  11. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
34 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  6. 2019Passage Re-ranking with BERTcited by 645
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  9. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  12. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  13. 2023Scaling Data-Constrained Language Modelscited by 403
  14. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  15. 2023Improving Text Embeddings with Large Language Modelscited by 303
  16. 2021Scaling Laws for Transfercited by 272
  17. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  18. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  19. 2024Generative Representational Instruction Tuningcited by 233
  20. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  21. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  22. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  23. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  24. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  25. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  26. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  27. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  28. 2024Making Text Embedders Few-Shot Learnerscited by 79
  29. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  30. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  31. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
  32. 2024Scaling Laws For Dense Retrievalcited by 29
  33. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  34. 2025Scaling Sparse and Dense Retrieval in Decoder-Only LLMscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Modelscited by 5
  2. 2026Scaling Laws for Embedding Dimension in Information Retrievalcited by 2