mlsanityData to 7 Oct 2026

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

arXiv 2205.12035 ↗First posted 24 May 2022cs.CLcited by 118

Not on the map. Why?

Cited by 20 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite pre-training's progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval. In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE). RetroMAE is highlighted by three critical designs. 1) A novel MAE workflow, where the input sentence is polluted for encoder and decoder with different masks. The sentence embedding is generated from the encoder's masked input; then, the original sentence is recovered based on the sentence embedding and the decoder's masked input via masked language modeling. 2) Asymmetric model structure, with a full-scale BERT like transformer as encoder, and a one-layer transformer as decoder. 3) Asymmetric masking ratios, with a moderate ratio for encoder: 15~30%, and an aggressive ratio for decoder: 50~70%. Our framework is simple to realize and empirically competitive: the pre-trained models dramatically improve the SOTA performances on a wide range of dense retrieval benchmarks, like BEIR and MS MARCO. The source code and pre-trained models are made publicly available at https://github.com/staoxiao/RetroMAE so as to inspire more interesting research.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

39 of 44 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
27 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  4. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  8. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  11. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  12. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  13. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  14. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  15. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  16. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  17. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  18. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  19. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  20. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  21. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  22. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  23. 2020TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrievalcited by 24
  24. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  25. 2021Training Large-Scale News Recommenders with Pretrained Language Models in the Loopcited by 18
  26. 2022Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Searchcited by 17
  27. 2022Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddingscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 24; 2024: 39; 2025: 37; 2026: 13.

118 papers.

Sort:

By area:

  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  4. 2024Generative Representational Instruction Tuningcited by 227
  5. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  6. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  7. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  8. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  9. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  10. 2024Making Text Embedders Few-Shot Learnerscited by 78
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
38 more
  1. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2023Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Trainingcited by 35
  4. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  5. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  6. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  7. 2024Scaling Laws For Dense Retrievalcited by 28
  8. 2025Provence: efficient and robust context pruning for retrieval-augmented generationcited by 25
  9. 2023Language Models As Semantic Indexerscited by 20
  10. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  11. 2024Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decodingcited by 19
  12. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  13. 2024Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integrationcited by 17
  14. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunitiescited by 17
  15. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  16. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15
  17. 2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
  18. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  19. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  20. 2024Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrievalcited by 13
  21. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  22. 2025Retrieval-augmented Large Language Models for Financial Time Series Forecastingcited by 12
  23. 2023Synergistic Interplay between Search and Large Language Models for Information Retrievalcited by 11
  24. 2024ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domaincited by 11
  25. 2025Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrievalcited by 11
  26. 2024ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labelingcited by 10
  27. 2025Efficient Constant-Space Multi-Vector Retrievalcited by 10
  28. 2023DAPR: A Benchmark on Document-Aware Passage Retrievalcited by 9
  29. 2024Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systemscited by 9
  30. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  31. 2024Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrieverscited by 8
  32. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  33. 2023Lexically-Accelerated Dense Retrievalcited by 7
  34. 2024Improve Dense Passage Retrieval with Entailment Tuningcited by 7
  35. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  36. 2025Granite Embedding Modelscited by 7
  37. 2025Universal Item Tokenization for Transferable Generative Recommendationcited by 7
  38. 2025Granite Embedding R2 Modelscited by 7

50 most cited of 118 All 118 (JSON)