mlsanityData to 7 Oct 2026

Query-as-context Pre-training for Dense Passage Retrieval

arXiv 2212.09598 ↗First posted 19 Dec 2022cs.IR cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the possibility of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains and meanwhile speeds up training, demonstrating its effectiveness and efficiency. Our code will be available at https://github.com/caskcsg/ir/tree/main/cotmae-qc .

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

44 of 51 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  10. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  11. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  12. 2020Overview of the TREC 2019 deep learning trackcited by 282
32 more
  1. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  2. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  3. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  4. 2021Overview of the TREC 2020 deep learning trackcited by 193
  5. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  6. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  7. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  8. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  9. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  10. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  11. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  12. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  13. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  14. 2020Learning Dense Representations of Phrases at Scalecited by 91
  15. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  16. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  17. 2021Few-Shot Conversational Dense Retrievalcited by 71
  18. 2021Learning Passage Impacts for Inverted Indexescited by 68
  19. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  20. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  21. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  22. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  23. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  24. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  25. 2021Adaptive Information Seeking for Open-Domain Question Answeringcited by 29
  26. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  27. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  28. 2022LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrievalcited by 18
  29. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  30. 2020Neural Passage Retrieval with Improved Negative Contrastcited by 11
  31. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  32. 2022Learning Diverse Document Representations with Deep Query Interactions for Dense Retrievalcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  2. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 15
  3. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  4. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  5. 2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
  6. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3
  7. 2024QAEncoder: Towards Aligned Representation Learning in Question Answering Systemscited by 3
  8. 2026Mem-π: Adaptive Memory through Learning When and What to Generatecited by 3
  9. 2024DocReLM: Mastering Document Retrieval with Language Modelcited by 0