mlsanityData to 7 Oct 2026

Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders

arXiv 2304.01016 ↗First posted 31 Mar 2023cs.CL cs.AI cs.IRcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders. First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual encoders in dense retrieval can lead to improved inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training. Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 34 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

31 of 34 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
19 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  6. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  8. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  10. 2021Learned Token Pruning for Transformerscited by 138
  11. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  12. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  13. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  14. 2019Scheduled Sampling for Transformerscited by 46
  15. 2020Least squares binary quantization of neural networkscited by 21
  16. 2022Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Poolingcited by 12
  17. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  18. 2021Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answeringcited by 8
  19. 2021DOCmT5: Document-Level Pretraining of Multilingual Language Modelscited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  2. 20242D Matryoshka Training for Information Retrievalcited by 6
  3. 2023Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiencycited by 5
  4. 2025LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representationscited by 4
  5. 2025LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs
  2. 2026EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering
  3. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  4. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  5. 2026OBLIQ-IR: Training a Dense Retriever for Oblique Queries ↗
  6. 2024A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks
  7. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
  8. 2025OSCAR: Online Soft Compression And Reranking