mlsanityData to 7 Oct 2026

CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval

arXiv 2304.03158 ↗First posted 5 Apr 2023cs.CL cs.AI cs.IRcited by 5

Not on the map. Why?

Abstract (arXiv)

Growing techniques have been emerging to improve the performance of passage retrieval. As an effective representation bottleneck pretraining technique, the contextual masked auto-encoder utilizes contextual embedding to assist in the reconstruction of passages. However, it only uses a single auto-encoding pre-task for dense representation pre-training. This study brings multi-view modeling to the contextual masked auto-encoder. Firstly, multi-view representation utilizes both dense and sparse vectors as multi-view representations, aiming to capture sentence semantics from different aspects. Moreover, multiview decoding paradigm utilizes both autoencoding and auto-regressive decoders in representation bottleneck pre-training, aiming to provide both reconstructive and generative signals for better contextual representation pretraining. We refer to this multi-view pretraining method as CoT-MAE v2. Through extensive experiments, we show that CoT-MAE v2 is effective and robust on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

36 of 46 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  11. 2020Overview of the TREC 2019 deep learning trackcited by 282
  12. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
24 more
  1. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  2. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  3. 2021Overview of the TREC 2020 deep learning trackcited by 193
  4. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  5. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  6. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  7. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  8. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  9. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  10. 2020Learning Dense Representations of Phrases at Scalecited by 91
  11. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  12. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  13. 2021Few-Shot Conversational Dense Retrievalcited by 71
  14. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  15. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  16. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46
  17. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  18. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  19. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  20. 2021Adaptive Information Seeking for Open-Domain Question Answeringcited by 29
  21. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  22. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  23. 2022LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrievalcited by 18
  24. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  2. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  3. 2023Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrievalcited by 7
  4. 2023CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrievalcited by 6
  5. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3