mlsanityData to 7 Oct 2026

Unified Language Model Pre-training for Natural Language Understanding and Generation

arXiv 1905.03197 ↗First posted 8 May 2019cs.CLcited by 668

Not on the map. Why?

Abstract (arXiv)

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.

Key facts. Cited by 668 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 183; 2022: 184; 2023: 138; 2024: 96; 2025: 48; 2026: 17.

668 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  12. 2021GPT Understands, Toocited by 715
38 more
  1. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  5. 2024Large Language Models: A Surveycited by 506
  6. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  9. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  11. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  12. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  14. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  15. 2022Large Language Models Are Reasoning Teacherscited by 318
  16. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  17. 2022UL2: Unifying Language Learning Paradigmscited by 292
  18. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  19. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  20. 2022Prompt Distribution Learningcited by 235
  21. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  22. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  23. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  24. 2021How Many Data Points is a Prompt Worth?cited by 197
  25. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  26. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  27. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  28. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  29. 2021The Values Encoded in Machine Learning Researchcited by 172
  30. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  31. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 166
  32. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  33. 2022A Review on Language Models as Knowledge Basescited by 156
  34. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  35. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  36. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  37. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  38. 2024Continual Learning for Large Language Models: A Surveycited by 144

50 most cited of 668 All 668 (JSON)