mlsanityData to 7 Oct 2026

Curriculum Learning for Dense Retrieval Distillation

arXiv 2204.13679 ↗First posted 28 Apr 2022cs.IR cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that more effective dense retrieval models can be obtained by distilling ranking knowledge from an existing base re-ranking model. In this paper, we propose a generic curriculum learning based optimization framework called CL-DRD that controls the difficulty level of training data produced by the re-ranking (teacher) model. CL-DRD iteratively optimizes the dense retrieval (student) model by increasing the difficulty of the knowledge distillation data made available to it. In more detail, we initially provide the student model coarse-grained preference pairs between documents in the teacher's ranking and progressively move towards finer-grained pairwise document ordering requirements. In our experiments, we apply a simple implementation of the CL-DRD framework to enhance two state-of-the-art dense retrieval models. Experiments on three public passage retrieval datasets demonstrate the effectiveness of our proposed framework.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

22 of 32 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  7. 2019Passage Re-ranking with BERTcited by 629
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2020Overview of the TREC 2019 deep learning trackcited by 282
  11. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  12. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
10 more
  1. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  2. 2017Teacher-Student Curriculum Learningcited by 197
  3. 2021Overview of the TREC 2020 deep learning trackcited by 193
  4. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  5. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  6. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  7. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  8. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  9. 2019Understanding the Behaviors of BERT in Rankingcited by 49
  10. 2020Training Curricula for Open Domain Answer Re-Rankingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 11; 2024: 9; 2025: 8; 2026: 4.

38 papers.

Sort:

By area:

  1. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  2. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  3. 2024SPLADE-v3: New baselines for SPLADEcited by 69
  4. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  5. 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
  6. 2024Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decodingcited by 19
  7. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  8. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  9. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  10. 2023Distillation from Heterogeneous Models for Top-K Recommendationcited by 13
  11. 2025Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generationcited by 13
  12. 2025APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Trainingcited by 12
26 more
  1. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  2. 2025Lightweight and Direct Document Relevance Optimization for Generative Information Retrievalcited by 9
  3. 2023Effective Matching of Patients to Clinical Trials using Entity Extraction and Neural Re-rankingcited by 8
  4. 2023Dense Retrieval Adaptation using Target Domain Descriptioncited by 8
  5. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  6. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  7. 2024HiCuLR: Hierarchical Curriculum Learning for Rhetorical Role Labeling of Legal Documentscited by 6
  8. 2023Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiencycited by 5
  9. 2023Noisy Pair Corrector for Dense Retrievalcited by 5
  10. 2022Adam: Dense Retrieval Distillation with Adaptive Dark Examplescited by 4
  11. 2024Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Modelcited by 4
  12. 2025Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevancecited by 4
  13. 2023Multivariate Representation Learning for Information Retrievalcited by 3
  14. 2024Distillation Enhanced Generative Retrievalcited by 2
  15. 2022LEAD: Liberal Feature-based Distillation for Dense Retrievalcited by 1
  16. 2024Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoderscited by 1
  17. 2025DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generationcited by 1
  18. 2023Mitigating Pooling Bias in E-commerce Search via False Negative Estimationcited by 0
  19. 2023Efficient Title Reranker for Fast and Improved Knowledge-Intense NLPcited by 0
  20. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  21. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  22. 2025Domain-Aware RAG: MoL-Enhanced RL for Efficient Training and Scalable Retrievalcited by 0
  23. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  24. 2026DSIRM: Learning Query-Bridged Discrete Semantic Identifiers for E-commerce Relevance Modelingcited by 0
  25. 2026Representation Curriculum: Stagewise Training for Robust Ranking and Allocationcited by 0
  26. 2026Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrievalcited by 0