mlsanityData to 7 Oct 2026

Condenser: a Pre-training Architecture for Dense Retrieval

arXiv 2104.08253 ↗First posted 16 Apr 2021cs.CL cs.IRcited by 162

Not on the map. Why?

Abstract (arXiv)

Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

38 of 53 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
26 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2018Universal Sentence Encodercited by 485
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  13. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  14. 2015Skip-Thought Vectorscited by 321
  15. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  16. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  17. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  18. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  19. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  20. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  21. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  22. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  23. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  24. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  25. 2020Efficient Document Re-Ranking for Transformers by Precomputing Term Representationscited by 44
  26. 2020DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modelingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 47; 2023: 48; 2024: 28; 2025: 24; 2026: 9.

162 papers.

Sort:

By area:

  1. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  6. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  8. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  9. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  10. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  11. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  12. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
38 more
  1. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  4. 2024DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Modelscited by 63
  5. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 61
  6. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  7. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  8. 2021Learning to Retrieve Passages without Supervisioncited by 46
  9. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  10. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  11. 2022Towards Robust Ranker for Text Retrievalcited by 37
  12. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  13. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  14. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  15. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  16. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  17. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  18. 2024Scaling Laws For Dense Retrievalcited by 28
  19. 2022Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrievalcited by 26
  20. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  21. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  22. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  23. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  24. 2024Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Modelscited by 20
  25. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  26. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  27. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  28. 2022CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrievalcited by 19
  29. 2023Chain-of-Skills: A Configurable Model for Open-domain Question Answeringcited by 19
  30. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  31. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  32. 2021You Only Need One Model for Open-domain Question Answeringcited by 17
  33. 2022Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Searchcited by 17
  34. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  35. 2022ConTextual Masked Auto-Encoder for Dense Passage Retrievalcited by 16
  36. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  37. 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
  38. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15

50 most cited of 162 All 162 (JSON)