mlsanityData to 7 Oct 2026

XLNet: Generalized Autoregressive Pretraining for Language Understanding

arXiv 1906.08237 ↗First posted 19 Jun 2019cs.CL cs.LGcited by 2,801

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

Key facts. Cited by 2,801 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 838; 2022: 645; 2023: 572; 2024: 414; 2025: 240; 2026: 90.

2,801 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
38 more
  1. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  2. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  4. 2021Ethical and social risks of harm from Language Modelscited by 957
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2021GPT Understands, Toocited by 715
  10. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  13. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  14. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  15. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  16. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  17. 2024Large Language Models: A Surveycited by 506
  18. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  19. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  20. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  21. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  22. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  23. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  24. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  25. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  26. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  27. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  28. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  29. 2022Multimodal Learning with Transformers: A Surveycited by 399
  30. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  31. 2021Pay Attention to MLPscited by 367
  32. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  33. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  34. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  35. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  36. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  37. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  38. 2022UL2: Unifying Language Learning Paradigmscited by 292

50 most cited of 2,801 All 2,801 (JSON)