mlsanityData to 13 Oct 2026

Large Dual Encoders Are Generalizable Retrievers

arXiv 2112.07899 ↗First posted 15 Dec 2021cs.IR cs.CLcited by 455

Not on the map. Why?

Cited by 77 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual encoder, where the final score is simply a dot-product between a query vector and a passage vector, is too limited to make dual encoders an effective retrieval model for out-of-domain generalization. In this paper, we challenge this belief by scaling up the size of the dual encoder model em while keeping the bottleneck embedding size fixed. With multi-stage training, surprisingly, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization. Experimental results show that our dual encoders, Generalizable T5-based dense Retrievers (GTR), outperform %ColBERT~citekhattab2020colbert and existing sparse and dense retrievers on the BEIR dataset~citethakur2021beir significantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10% of MS Marco supervised data to achieve the best out-of-domain performance. All the GTR models are released at https://tfhub.dev/google/collections/gtr/1.

Key facts. Cited by 455 papers in the map (data through 13 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

28 of 41 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  5. 2017Billion-scale similarity search with GPUscited by 2,490
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,600
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,237
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,169
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 900
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 847
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 622
16 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 507
  2. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 468
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  4. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 412
  5. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  6. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  7. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  8. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  9. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
  10. 2019Learning Dense Representations for Entity Retrievalcited by 122
  11. 2020Open-Retrieval Conversational Question Answeringcited by 99
  12. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  13. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  14. 2018End-to-End Retrieval in Continuous Spacecited by 46
  15. 2019ReQA: An Evaluation for End-to-End Answer Retrieval Modelscited by 16
  16. 2020Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generationcited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 1; 2022: 38; 2023: 95; 2024: 152; 2025: 107; 2026: 62.

455 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,815
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,229
  4. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 790
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 714
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 697
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 630
  8. 2023Enabling Large Language Models to Generate Text with Citationscited by 588
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 562
  10. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 510
  11. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 467
  12. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 398
38 more
  1. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 364
  2. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  3. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 330
  4. 2023Improving Text Embeddings with Large Language Modelscited by 297
  5. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  6. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 275
  7. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 263
  8. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  9. 2024Generative Representational Instruction Tuningcited by 229
  10. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 226
  11. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 224
  12. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  13. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  14. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  15. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 149
  16. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 132
  17. 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 126
  18. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  19. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 122
  20. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  21. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 117
  22. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  23. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 109
  24. 2022Task-aware Retrieval with Instructionscited by 107
  25. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  26. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  27. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  28. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 93
  29. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  30. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 81
  31. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  32. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  33. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  34. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  35. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  36. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  37. 2024Repetition Improves Language Model Embeddingscited by 71
  38. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70

50 most cited of 455 All 455 (JSON)