# Scaling Sparse and Dense Retrieval in Decoder-Only LLMs

arXiv: https://arxiv.org/abs/2502.15526 · HTML: https://mlsanity.com/paper/2502.15526/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2025.

## Built on by (most cited first, 12 of 12)

- [Benchmarking Information Retrieval Models on Complex Retrieval Tasks](https://mlsanity.com/paper/2509.07253/) (2025)
- [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026)
- [Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models](https://mlsanity.com/paper/2511.07581/) (2025)
- [LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum](https://mlsanity.com/paper/2601.01684/) (2026)
- [An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc](https://mlsanity.com/paper/2505.15070/) (2025)
- [CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training](https://mlsanity.com/paper/2506.10844/) (2025)
- [Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs](https://mlsanity.com/paper/2508.17400/) (2025)
- [Scaling Laws for Embedding Dimension in Information Retrieval](https://mlsanity.com/paper/2602.05062/) (2026)
- [On the Challenges and Opportunities of Learned Sparse Retrieval for Code](https://mlsanity.com/paper/2603.22008/) (2026)
- [To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval](https://mlsanity.com/paper/2601.17500/) (2026)
- [The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles](https://mlsanity.com/paper/2605.01407/) (2026)
- [uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking](https://mlsanity.com/paper/2606.11945/) (2026)

## Abstract (arXiv)

Scaling large language models (LLMs) has shown great potential for improving retrieval model performance; however, previous studies have mainly focused on dense retrieval trained with contrastive loss (CL), neglecting the scaling behavior of other retrieval paradigms and optimization techniques, such as sparse retrieval and knowledge distillation (KD). In this work, we conduct a systematic comparative study on how different retrieval paradigms (sparse vs. dense) and fine-tuning objectives (CL vs. KD vs. their combination) affect retrieval performance across different model scales. Using MSMARCO passages as the training dataset, decoder-only LLMs (Llama-3 series: 1B, 3B, 8B), and a fixed compute budget, we evaluate various training configurations on both in-domain (MSMARCO, TREC DL) and out-of-domain (BEIR) benchmarks. Our key findings reveal that: (1) Scaling behaviors emerge clearly only with CL, where larger models achieve significant performance gains, whereas KD-trained models show minimal improvement, performing similarly across the 1B, 3B, and 8B scales. (2) Sparse retrieval models consistently outperform dense retrieval across both in-domain (MSMARCO, TREC DL) and out-of-domain (BEIR) benchmarks, and they demonstrate greater robustness to imperfect supervised signals. (3) We successfully scale sparse retrieval models with the combination of CL and KD losses at 8B scale, achieving state-of-the-art (SOTA) results in all evaluation sets.