Data to 13 Oct 2026

Learning Dense Representations of Phrases at Scale

arXiv 2012.12624 ↗First posted 23 Dec 2020cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

Key facts. Cited by 91 papers in the map (data through 13 Oct 2026). First posted on arXiv 23 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 16; 2022: 33; 2023: 13; 2024: 17; 2025: 8; 2026: 4.

91 papers.

Sort:

By area:

2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,229
2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 468
2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
2022Re2G: Retrieve, Rerank, Generatecited by 102
2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90

38 more

2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 61
2021Learning to Retrieve Passages without Supervisioncited by 46
2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
2022Nonparametric Masked Language Modelingcited by 40
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
2024Block-Attention for Efficient Prefillingcited by 35
2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
2022UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Miningcited by 33
2022Fine-Grained Distillation for Long Document Retrievalcited by 29
2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 29
2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
2023Copy Is All You Needcited by 25
2021Robust Retrieval Augmented Generation for Zero-shot Slot Fillingcited by 24
2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
2022A Survey for Efficient Open Domain Question Answeringcited by 20
2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19
2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
2022Exploring Dual Encoder Architectures for Question Answeringcited by 19
2022Saving Dense Retriever from Shortcut Dependency in Conversational Searchcited by 18
2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
2022ConTextual Masked Auto-Encoder for Dense Passage Retrievalcited by 16
2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
2025Episodic Memories Generation and Evaluation Benchmark for Large Language Modelscited by 14
2021Graph-free Multi-hop Reading Comprehension: A Select-to-Guide Strategycited by 13
2021Sentence-aware Contrastive Learning for Open-Domain Passage Retrievalcited by 11
2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
2024Retrieval is Accurate Generationcited by 10
2022Query-as-context Pre-training for Dense Passage Retrievalcited by 9
2024textttMixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularitycited by 9
2024Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselinescited by 9
2023SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examplescited by 8
2023Generator-Retriever-Generator Approach for Open-Domain Question Answeringcited by 8
2022A Study on the Efficiency and Generalization of Light Hybrid Retrieverscited by 7

50 most cited of 91 All 91 (JSON)