mlsanityData to 7 Oct 2026

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

arXiv 2103.10360 ↗First posted 18 Mar 2021cs.CL cs.AI cs.LGcited by 1,026

Not on the map. Why?

Cited by 115 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

Key facts. Cited by 1,026 papers in the map (data through 7 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

37 of 64 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
25 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  9. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2019Text Summarization with Pretrained Encoderscited by 553
  13. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  14. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  15. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  16. 2018A Simple Method for Commonsense Reasoningcited by 223
  17. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  18. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  19. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  20. 2020Enabling Language Models to Fill in the Blankscited by 121
  21. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  22. 2020Blank Language Modelscited by 43
  23. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42
  24. 2019Text Infillingcited by 32
  25. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 17; 2023: 321; 2024: 530; 2025: 121; 2026: 35.

1,026 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
38 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  13. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  15. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  16. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  17. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  19. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  20. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  21. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  22. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  23. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  24. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  25. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  26. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  27. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  28. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  29. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  30. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  31. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  32. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  33. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  34. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  35. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  36. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  37. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  38. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243

50 most cited of 1,026 All 1,026 (JSON)