mlsanityData to 7 Oct 2026

ConTextual Masked Auto-Encoder for Dense Passage Retrieval

arXiv 2208.07670 ↗First posted 16 Aug 2022cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent studies have explored improving pre-trained language models to boost dense retrieval performance. This paper proposes CoT-MAE (ConTextual Masked Auto-Encoder), a simple yet effective generative pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric encoder-decoder architecture that learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. Precisely, self-supervised masked auto-encoding learns to model the semantics of the tokens inside a text span, and context-supervised masked auto-encoding learns to model the semantical correlation between the text spans. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines, demonstrating the high efficiency of CoT-MAE. Our code is available at https://github.com/caskcsg/ir/tree/main/cotmae.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 16 Aug 2022.

Builds on

35 of 43 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  11. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  12. 2020Overview of the TREC 2019 deep learning trackcited by 282
23 more
  1. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  2. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  3. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  4. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  5. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  6. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  7. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  8. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  9. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  10. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  11. 2020Learning Dense Representations of Phrases at Scalecited by 91
  12. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  13. 2021Few-Shot Conversational Dense Retrievalcited by 71
  14. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  15. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  16. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  17. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  18. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46
  19. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  20. 2021Adaptive Information Seeking for Open-Domain Question Answeringcited by 29
  21. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  22. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  23. 2020Neural Passage Retrieval with Improved Negative Contrastcited by 11

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 8; 2024: 4; 2025: 2.

16 papers.

Sort:

By area:

  1. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  2. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  3. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  4. 2024Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systemscited by 9
  5. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  6. 2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
  7. 2022Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrievalcited by 4
  8. 2023Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularizationcited by 4
  9. 2025CIMAGE: Exploiting the Conditional Independence in Masked Graph Auto-encoderscited by 4
  10. 2023Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrievalcited by 3
  11. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3
  12. 2023Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systemscited by 2
4 more
  1. 2023PUNR: Pre-training with User Behavior Modeling for News Recommendationcited by 1
  2. 2023Efficient Title Reranker for Fast and Improved Knowledge-Intense NLPcited by 0
  3. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  4. 2025M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformercited by 0