mlsanityData to 7 Oct 2026

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

arXiv 2002.12804 ↗First posted 28 Feb 2020cs.CLcited by 204

Not on the map. Why?

Abstract (arXiv)

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 66; 2022: 63; 2023: 40; 2024: 25; 2025: 7; 2026: 3.

204 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  5. 2021Video Swin Transformercited by 1,090
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
38 more
  1. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  2. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  3. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  4. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  5. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  6. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  7. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  10. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  11. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  12. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  13. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  14. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  15. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  16. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  17. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  18. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  19. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  20. 2022Language Models are General-Purpose Interfacescited by 96
  21. 2021CAT: Cross Attention in Vision Transformercited by 92
  22. 2021M6: A Chinese Multimodal Pretrainercited by 91
  23. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  24. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  25. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  26. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  27. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  28. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  29. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  30. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  31. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  32. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  33. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  34. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  35. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  36. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  37. 2021SeMask: Semantically Masked Transformers for Semantic Segmentationcited by 55
  38. 2022Your Transformer May Not be as Powerful as You Expectcited by 55

50 most cited of 204 All 204 (JSON)