mlsanityData to 7 Oct 2026

MPNet: Masked and Permuted Pre-training for Language Understanding

arXiv 2004.09297 ↗First posted 20 Apr 2020cs.CL cs.LGcited by 973

Not on the map. Why?

Cited by 55 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet.

Key facts. Cited by 973 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 95; 2023: 182; 2024: 239; 2025: 224; 2026: 206.

973 papers.

Sort:

By area:

  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  3. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  6. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  11. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  12. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
38 more
  1. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  2. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  3. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  4. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  5. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  6. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  7. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  8. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  9. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  10. 2023PRODIGY: Enabling In-context Learning Over Graphscited by 97
  11. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  12. 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
  13. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  14. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  15. 2023Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentencecited by 81
  16. 2021Diverse Image Inpainting with Bidirectional and Autoregressive Transformerscited by 80
  17. 2024GTA: A Benchmark for General Tool Agentscited by 77
  18. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  19. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  20. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  21. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  22. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  23. 2023Locally Differentially Private Document Generation Using Zero Shot Promptingcited by 63
  24. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 62
  25. 2022Legal Prompting: Teaching a Language Model to Think Like a Lawyercited by 60
  26. 2023In-context Example Selection with Influencescited by 60
  27. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 58
  28. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  29. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  30. 2022I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classificationcited by 54
  31. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  32. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  33. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  34. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 49
  35. 2024Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agentscited by 49
  36. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  37. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 48
  38. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 47

50 most cited of 973 All 973 (JSON)