mlsanityData to 14 Oct 2026

PROD: Progressive Distillation for Dense Retrieval

arXiv 2209.13335 ↗First posted 27 Sep 2022cs.IR cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student. However, this expectation does not always come true. It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student. To bridge the gap, we propose PROD, a PROgressive Distillation method, for dense retrieval. PROD consists of a teacher progressive distillation and a data progressive distillation to gradually improve the student. We conduct extensive experiments on five widely-used benchmarks, MS MARCO Passage, TREC Passage 19, TREC Document 19, MS MARCO Document and Natural Questions, where PROD achieves the state-of-the-art within the distillation methods for dense retrieval. The code and models will be released.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

49 of 58 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2016Learning without Forgettingcited by 2,496
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  11. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
37 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  3. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  6. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  7. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  8. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  9. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  10. 2020Overview of the TREC 2019 deep learning trackcited by 286
  11. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  12. 2019Document Expansion by Query Predictioncited by 231
  13. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  14. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  15. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  16. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  17. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  18. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  19. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
  20. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  21. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 113
  22. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 83
  23. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  24. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  25. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  26. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  27. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  28. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  29. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  30. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  31. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  32. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  33. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 21
  34. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 17
  35. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  36. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compressioncited by 8
  37. 2022Unified and Effective Ensemble Knowledge Distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 5; 2025: 5; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  2. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  3. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  4. 2025Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generationcited by 13
  5. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  6. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  7. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  8. 2023Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiencycited by 5
  9. 2022Adam: Dense Retrieval Distillation with Adaptive Dark Examplescited by 4
  10. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  11. 2023Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularizationcited by 4
  12. 2024PairDistill: Pairwise Relevance Distillation for Dense Retrievalcited by 3
9 more
  1. 2025LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBaycited by 2
  2. 2022LEAD: Liberal Feature-based Distillation for Dense Retrievalcited by 1
  3. 2026MemAdapter: Fast Alignment across Agent Memory Paradigms via Generative Subgraph Retrievalcited by 1
  4. 2023On Elastic Language Modelscited by 0
  5. 2023Efficient Title Reranker for Fast and Improved Knowledge-Intense NLPcited by 0
  6. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  7. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  8. 2025MURR: Model Updating with Regularized Replay for Searching a Document Streamcited by 0
  9. 2025Efficient Speech Watermarking for Speech Synthesis via Progressive Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
  2. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  3. 2025Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective
  4. 2022Extreme compression of sentence-transformer ranker models: faster inference, longer battery life, and less storage on edge devices
  5. 2023Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval
  6. 2024Dense Passage Retrieval: Is it Retrieving?
  7. 2022Fine-Grained Distillation for Long Document Retrieval
  8. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling