mlsanityData to 14 Oct 2026

Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling

arXiv 2104.06967 ↗First posted 14 Apr 2021cs.IR cs.CLcited by 238

Not on the map. Why?

Abstract (arXiv)

A vital step towards the widespread adoption of neural retrieval models is their resource efficiency throughout the training, indexing and query workflows. The neural IR community made great advancements in training effective dual-encoder dense retrieval (DR) models recently. A dense text retrieval model uses a single vector representation per query and passage to score a match, which enables low-latency first stage retrieval with a nearest neighbor search. Increasingly common, training approaches require enormous compute power, as they either conduct negative passage sampling out of a continuously updating refreshing index or require very large batch sizes for in-batch negative sampling. Instead of relying on more compute capability, we introduce an efficient topic-aware query and balanced margin sampling technique, called TAS-Balanced. We cluster queries once before training and sample queries out of a cluster per batch. We train our lightweight 6-layer DR model with a novel dual-teacher supervision that combines pairwise and in-batch negative teachers. Our method is trainable on a single consumer-grade GPU in under 48 hours (as opposed to a common configuration of 8x V100s). We show that our TAS-Balanced training method achieves state-of-the-art low-latency (64ms per query) results on two TREC Deep Learning Track query sets. Evaluated on NDCG@10, we outperform BM25 by 44%, a plainly trained DR by 19%, docT5query by 11%, and the previous best DR model by 5%. Additionally, TAS-Balanced produces the first dense retriever that outperforms every other method on recall at any cutoff on TREC-DL and allows more resource intensive re-ranking models to operate on fewer passages to improve results further.

Key facts. Cited by 238 papers in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

32 of 49 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
20 more
  1. 2019Passage Re-ranking with BERTcited by 645
  2. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  3. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  4. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  5. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  6. 2021Overview of the TREC 2020 deep learning trackcited by 196
  7. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  8. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  9. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  10. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  11. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  12. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60
  13. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  14. 2020TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrievalcited by 24
  15. 2020Understanding BERT Rankers Under Distillationcited by 15
  16. 2020Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficientlycited by 11
  17. 2019Let's measure run time! Extending the IR replicability infrastructure to include performance aspectscited by 10
  18. 2020Distilling Knowledge for Fast Retrieval-based Chat-botscited by 10
  19. 2020Training Curricula for Open Domain Answer Re-Rankingcited by 10
  20. 2020Simplified TinyBERT: Knowledge Distillation for Document Retrievalcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 55; 2023: 47; 2024: 52; 2025: 37; 2026: 28.

238 papers.

Sort:

By area:

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  6. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  8. 2024Tool Learning with Large Language Models: A Surveycited by 223
  9. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  10. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  11. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  12. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
38 more
  1. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
  2. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  3. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  4. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  5. 2022Task-aware Retrieval with Instructionscited by 107
  6. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  7. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  8. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  9. 2024What Evidence Do Language Models Find Convincing?cited by 75
  10. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 70
  11. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  12. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  13. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 64
  14. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  15. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  16. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 60
  17. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  18. 2021Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedbackcited by 47
  19. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  20. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  21. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  22. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  23. 2022NGAME: Negative Mining-aware Mini-batching for Extreme Classificationcited by 32
  24. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 31
  25. 2024INMS: Memory Sharing for Large Language Model based Agentscited by 31
  26. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  27. 2024Scaling Laws For Dense Retrievalcited by 29
  28. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  29. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  30. 2022An Efficiency Study for SPLADE Modelscited by 26
  31. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Modelscited by 26
  32. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 25
  33. 2021A Proposed Conceptual Framework for a Representational Approach to Information Retrievalcited by 24
  34. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  35. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  36. 2025Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunitiescited by 22
  37. 2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
  38. 2022A Thorough Examination on Zero-shot Dense Retrievalcited by 21

50 most cited of 238 All 238 (JSON)