mlsanityData to 14 Oct 2026

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

arXiv 2205.09153 ↗First posted 18 May 2022cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Neural retrievers based on pre-trained language models (PLMs), such as dual-encoders, have achieved promising performance on the task of open-domain question answering (QA). Their effectiveness can further reach new state-of-the-arts by incorporating cross-architecture knowledge distillation. However, most of the existing studies just directly apply conventional distillation methods. They fail to consider the particular situation where the teacher and student have different structures. In this paper, we propose a novel distillation method that significantly advances cross-architecture distillation for dual-encoders. Our method 1) introduces a self on-the-fly distillation method that can effectively distill late interaction (i.e., ColBERT) to vanilla dual-encoder, and 2) incorporates a cascade distillation process to further improve the performance with a cross-encoder teacher. Extensive experiments are conducted to validate that our proposed solution outperforms strong baselines and establish a new state-of-the-art on open-domain QA benchmarks.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

46 of 54 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
34 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  6. 2019Passage Re-ranking with BERTcited by 645
  7. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  8. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  10. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  11. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  12. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  13. 2019Document Expansion by Query Predictioncited by 231
  14. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  15. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  16. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  17. 2019Multi-Stage Document Ranking with BERTcited by 201
  18. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  19. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  20. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  21. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  22. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  23. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 113
  24. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  25. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 83
  26. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  27. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  28. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  29. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  30. 2021Pre-trained Language Model for Web-scale Retrieval in Baidu Searchcited by 31
  31. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  32. 2020Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentationcited by 18
  33. 2021Improving Bi-encoder Document Ranking Models with Two Rankers and Multi-teacher Distillationcited by 7
  34. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 17; 2023: 11; 2024: 7; 2025: 6.

41 papers.

Sort:

By area:

  1. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  2. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  3. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 48
  4. 2022Towards Robust Ranker for Text Retrievalcited by 37
  5. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  6. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  7. 2022PROD: Progressive Distillation for Dense Retrievalcited by 21
  8. 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
  9. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 20
  10. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  11. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  12. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
29 more
  1. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  2. 2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11
  3. 2024Large Language Models for Relevance Judgment in Product Searchcited by 8
  4. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  5. 2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
  6. 2023CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrievalcited by 5
  7. 2025Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modelingcited by 5
  8. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  9. 2022Learning Diverse Document Representations with Deep Query Interactions for Dense Retrievalcited by 4
  10. 2022Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrievalcited by 4
  11. 2022Adam: Dense Retrieval Distillation with Adaptive Dark Examplescited by 4
  12. 2023Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularizationcited by 4
  13. 2023Plot Retrieval as an Assessment of Abstract Semantic Associationcited by 3
  14. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3
  15. 2024UniRetriever: Multi-task Candidates Selection for Various Context-Adaptive Conversational Retrievalcited by 3
  16. 2024Learning label-label correlations in Extreme Multi-label Classification via Label Featurescited by 3
  17. 2025ListConRanker: A Contrastive Text Reranker with Listwise Encodingcited by 3
  18. 2022Hybrid Inverted Index Is a Robust Accelerator for Dense Retrievalcited by 2
  19. 2022RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 2
  20. 2025Can Cross Encoders Produce Useful Sentence Embeddings?cited by 2
  21. 2025LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBaycited by 2
  22. 2022LEAD: Liberal Feature-based Distillation for Dense Retrievalcited by 1
  23. 2023Towards Robust Text Retrieval with Progressive Learningcited by 1
  24. 2023Event-driven Real-time Retrieval in Web Searchcited by 1
  25. 2025LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inferencecited by 1
  26. 2023Hybrid Retrieval and Multi-stage Text Ranking Solution at TREC 2022 Deep Learning Trackcited by 0
  27. 2023Mitigating Pooling Bias in E-commerce Search via False Negative Estimationcited by 0
  28. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  29. 2024QABISAR: Query-Article Bipartite Interactions for Statutory Article Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation
  2. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  3. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation
  4. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
  5. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach
  6. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  7. 2024How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
  8. 2021Improving Question Answering Performance Using Knowledge Distillation and Active Learning