mlsanityData to 7 Oct 2026

Pre-training via Paraphrasing

arXiv 2006.15020 ↗First posted 26 Jun 2020cs.CL cs.LG stat.MLcited by 92

Not on the map. Why?

Abstract (arXiv)

We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 47; 2022: 21; 2023: 6; 2024: 10; 2025: 8.

92 papers.

Sort:

By area:

  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  4. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  5. 2022Memorizing Transformerscited by 219
  6. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  7. 2022A Survey on Retrieval-Augmented Text Generationcited by 157
  8. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  9. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  10. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  11. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  12. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
38 more
  1. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  2. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  3. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  4. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  5. 2021Neural Machine Translation with Monolingual Translation Memorycited by 61
  6. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  7. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  8. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  9. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  10. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  11. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  12. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  13. 2021Fast Nearest Neighbor Machine Translationcited by 37
  14. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  15. 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
  16. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  17. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  18. 2022OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answeringcited by 29
  19. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  20. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  21. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  22. 2021Learning Opinion Summarizers by Selecting Informative Reviewscited by 25
  23. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  24. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  25. 2021ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generationcited by 23
  26. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  27. 2021Self-Supervised Test-Time Learning for Reading Comprehensioncited by 22
  28. 2021RetGen: A Joint framework for Retrieval and Grounded Text Generation Modelingcited by 22
  29. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  30. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  31. 2024Revisiting the Robustness of Watermarking to Paraphrasing Attackscited by 19
  32. 2022BARTSmiles: Generative Masked Language Models for Molecular Representationscited by 18
  33. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  34. 2021CDLM: Cross-Document Language Modelingcited by 17
  35. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  36. 2021RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsingcited by 15
  37. 2021Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchycited by 14
  38. 2024Memorization in deep learning: A surveycited by 14

50 most cited of 92 All 92 (JSON)