mlsanityData to 7 Oct 2026

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

arXiv 2205.04733 ↗First posted 10 May 2022cs.IR cs.CLcited by 98

Not on the map. Why?

Abstract (arXiv)

Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

26 of 32 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  11. 2020Overview of the TREC 2019 deep learning trackcited by 282
  12. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
14 more
  1. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  2. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  3. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  4. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  5. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  6. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  7. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  8. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  9. 2021Learning Passage Impacts for Inverted Indexescited by 68
  10. 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
  11. 2021Learning to Retrieve Passages without Supervisioncited by 46
  12. 2020Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficientlycited by 10
  13. 2021Match Your Words! A Study of Lexical Matching in Neural Information Retrievalcited by 10
  14. 2021A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 17; 2024: 30; 2025: 22; 2026: 23.

98 papers.

Sort:

By area:

  1. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  2. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  3. 2024SPLADE-v3: New baselines for SPLADEcited by 69
  4. 2022When Language Model Meets Private Librarycited by 54
  5. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  6. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  7. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  8. 2023NevIR: Negation in Neural Information Retrievalcited by 28
  9. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  10. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  11. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  12. 2024Results of the Big ANN: NeurIPS'23 competitioncited by 21
38 more
  1. 2024Ranked List Truncation for Large Language Model-based Re-Rankingcited by 20
  2. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  3. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  4. 2024ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrievalcited by 18
  5. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  6. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  7. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  8. 2025Lost at the Beginning of Reasoningcited by 15
  9. 2024PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methodscited by 14
  10. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12
  11. 2024Taxonomy-guided Semantic Indexing for Academic Paper Searchcited by 12
  12. 2023BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrievalcited by 11
  13. 2024Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomycited by 11
  14. 2023A Study on FGSM Adversarial Training for Neural Retrievalcited by 10
  15. 2023CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answeringcited by 10
  16. 2025CSPLADE: Learned Sparse Retrieval with Causal Language Modelscited by 10
  17. 2023DAPR: A Benchmark on Document-Aware Passage Retrievalcited by 9
  18. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspectivecited by 9
  19. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  20. 2024Mistral-SPLADE: LLMs for better Learned Sparse Retrievalcited by 8
  21. 2024DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Searchcited by 8
  22. 2025AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Rerankingcited by 8
  23. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  24. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  25. 2025GRAM: Generative Recommendation via Semantic-aware Multi-granular Late Fusioncited by 7
  26. 2022Zero-Shot Retrieval with Search Agents and Hybrid Environmentscited by 6
  27. 2024LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embeddingcited by 6
  28. 2024HyQE: Ranking Contexts with Hypothetical Query Embeddingscited by 6
  29. 2025Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generationcited by 6
  30. 2025Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansioncited by 6
  31. 2025RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedbackcited by 5
  32. 2026LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculumcited by 5
  33. 2026Learning Retrieval Models with Sparse Autoencoderscited by 5
  34. 2026Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agentscited by 5
  35. 2023Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrievalcited by 4
  36. 2023Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrievalcited by 4
  37. 2023PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Modelscited by 4
  38. 2024An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Rerankingcited by 4

50 most cited of 98 All 98 (JSON)