mlsanityData to 7 Oct 2026

Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency

arXiv 2306.11550 ↗First posted 5 Jun 2023cs.IR cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The information retrieval community has made significant progress in improving the efficiency of Dual Encoder (DE) dense passage retrieval systems, making them suitable for latency-sensitive settings. However, many proposed procedures are often too complex or resource-intensive, which makes it difficult for practitioners to adopt them or identify sources of empirical gains. Therefore, in this work, we propose a trivially simple recipe to serve as a baseline method for boosting the efficiency of DE retrievers leveraging an asymmetric architecture. Our results demonstrate that even a 2-layer, BERT-based query encoder can still retain 92.5% of the full DE performance on the BEIR benchmark via unsupervised distillation and proper student initialization. We hope that our findings will encourage the community to re-evaluate the trade-offs between method complexity and performance improvements.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

36 of 55 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
24 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2018Universal Sentence Encodercited by 485
  3. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  5. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  6. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  9. 2020Characterising Bias in Compressed Modelscited by 162
  10. 2020The Hardware Lotterycited by 160
  11. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  12. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  13. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  14. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  15. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  16. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  17. 2018Approximate Nearest Neighbor Search in High Dimensionscited by 45
  18. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  19. 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
  20. 2022Exploring Dual Encoder Architectures for Question Answeringcited by 19
  21. 2020Flexible retrieval with NMSLIB and FlexNeuARTcited by 14
  22. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  23. 2023Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoderscited by 5
  24. 2022Distribution-Aligned Fine-Tuning for Efficient Neural Retrievalcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
  2. 2025LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inferencecited by 1
  3. 2026Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Searchcited by 0
  4. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0
  5. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0