LEAD: Liberal Feature-based Distillation for Dense Retrieval
On the map: Knowledge distillation → Knowledge distillation for LLMs
Abstract (arXiv)
Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model. Traditional methods include response-based methods and feature-based methods. Response-based methods are widely used but suffer from lower upper limits of performance due to their ignorance of intermediate signals, while feature-based methods have constraints on vocabularies, tokenizers and model architectures. In this paper, we propose a liberal feature-based distillation method (LEAD). LEAD aligns the distribution between the intermediate layers of teacher model and student model, which is effective, extendable, portable and has no requirements on vocabularies, tokenizers, or model architectures. Extensive experiments show the effectiveness of LEAD on widely-used benchmarks, including MS MARCO Passage Ranking, TREC 2019 DL Track, MS MARCO Document Ranking and TREC 2020 DL Track. Our code is available in https://github.com/microsoft/SimXNS/tree/main/LEAD.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 43 of its 67 references that are in the map. First posted on arXiv 10 Dec 2022.
Builds on
43 of 67 references.
Closest ancestors · 1
Same direction, newest first
Other references · 42
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2020Language Models are Few-Shot Learnerscited by 32,320
- 2015Distilling the Knowledge in a Neural Networkcited by 10,180
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
- 2014FitNets: Hints for Thin Deep Netscited by 1,631
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,600
- 2020Knowledge Distillation: A Surveycited by 1,590
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,169
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,067
- 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 847
30 more
- 2019Improved Knowledge Distillation via Teacher Assistantcited by 569
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 504
- 2021Large Dual Encoders Are Generalizable Retrieverscited by 455
- 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 412
- 2020Overview of the TREC 2019 deep learning trackcited by 285
- 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
- 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
- 2019Document Expansion by Query Predictioncited by 226
- 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
- 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 211
- 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
- 2021Overview of the TREC 2020 deep learning trackcited by 195
- 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
- 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
- 2021Towards Understanding Knowledge Distillationcited by 158
- 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
- 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 144
- 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
- 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
- 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
- 2020Understanding and Improving Knowledge Distillationcited by 83
- 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
- 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
- 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
- 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
- 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
- 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
- 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
- 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
Built on by
Citing papers per year: 2023: 1.
1 paper.