mlsanityData to 14 Oct 2026

Topic-DPR: Topic-based Prompts for Dense Passage Retrieval

arXiv 2310.06626 ↗First posted 10 Oct 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Prompt-based learning's efficacy across numerous natural language processing tasks has led to its integration into dense passage retrieval. Prior research has mainly focused on enhancing the semantic understanding of pre-trained language models by optimizing a single vector as a continuous prompt. This approach, however, leads to a semantic space collapse; identical semantic information seeps into all representations, causing their distributions to converge in a restricted region. This hinders differentiation between relevant and irrelevant passages during dense retrieval. To tackle this issue, we present Topic-DPR, a dense passage retrieval model that uses topic-based prompts. Unlike the single prompt method, multiple topic-based prompts are established over a probabilistic simplex and optimized simultaneously through contrastive learning. This encourages representations to align with their topic distributions, improving space uniformity. Furthermore, we introduce a novel positive and negative sampling strategy, leveraging semi-structured data to boost dense retrieval efficiency. Experimental results from two datasets affirm that our method surpasses previous state-of-the-art retrieval techniques.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
18 more
  1. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  3. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  4. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  5. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  7. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  8. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  9. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 113
  10. 2019On the Use of ArXiv as a Datasetcited by 103
  11. 2020Learning Dense Representations of Phrases at Scalecited by 91
  12. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  13. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  14. 2022Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrieverscited by 18
  15. 2022PromptEM: Prompt-tuning for Low-resource Generalized Entity Matchingcited by 10
  16. 2022DPTDR: Deep Prompt Tuning for Dense Passage Retrievalcited by 10
  17. 2022LIDER: An Efficient High-dimensional Learned Index for Large-scale Dense Passage Retrievalcited by 8
  18. 2023Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representationscited by 2

Built on by

Not cited yet.