mlsanityData to 14 Oct 2026

Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

arXiv 2102.08036 ↗First posted 16 Feb 2021cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Recent years have seen a proliferation of attention mechanisms and the rise of Transformers in Natural Language Generation (NLG). Previously, state-of-the-art NLG architectures such as RNN and LSTM ran into vanishing gradient problems; as sentences grew larger, distance between positions remained linear, and sequential computation hindered parallelization since sentences were processed word by word. Transformers usher in a new era. In this paper, we explore three major Transformer-based models, namely GPT, BERT, and XLNet, that carry significant implications for the field. NLG is a burgeoning area that is now bolstered with rapid developments in attention mechanisms. From poetry generation to summarization, text generation derives benefit as Transformer-based language models achieve groundbreaking results.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

19 of 33 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
7 more
  1. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  2. 2015Listen, Attend and Spellcited by 543
  3. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  4. 2019An Attentive Survey of Attention Modelscited by 124
  5. 2019Attention in Natural Language Processingcited by 101
  6. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  7. 2019DocBERT: BERT for Document Classificationcited by 68

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 11; 2024: 4; 2025: 0; 2026: 2.

27 papers.

Sort:

By area:

  1. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 125
  2. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  3. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
  4. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  5. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  6. 2024Emergency Department Decision Support using Clinical Pseudo-notescited by 22
  7. 2022Generative Transformers for Design Concept Generationcited by 21
  8. 2021Generative Pre-Trained Transformer for Design Concept Generation: An Explorationcited by 17
  9. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  10. 2023Autonomous Agents in Software Development: A Vision Papercited by 12
  11. 2022Vision Transformers in Medical Imaging: A Reviewcited by 8
  12. 2023AE-GPT: Using Large Language Models to Extract Adverse Events from Surveillance Reports-A Use Case with Influenza Vaccine Adverse Eventscited by 8
15 more
  1. 2022Artificial Text Detection with Multiple Training Strategiescited by 4
  2. 2023Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?cited by 4
  3. 2023MultiNet with Transformers: A Model for Cancer Diagnosis Using Imagescited by 3
  4. 2021Textual Data Distributions: Kullback Leibler Textual Distributions Contrasts on GPT-2 Generated Texts, with Supervised, Unsupervised Learning on Vaccine & Market Topics & Sentimentcited by 2
  5. 2024Beyond Generative Artificial Intelligence: Roadmap for Natural Language Generationcited by 2
  6. 2026Multi-hop Reasoning and Retrieval in Embedding Space: Leveraging Large Language Models with Knowledgecited by 2
  7. 2021What's in a Measurement? Using GPT-3 on SemEval 2021 Task 8 -- MeasEvalcited by 1
  8. 2023More for Less: Compact Convolutional Transformers Enable Robust Medical Image Classification with Limited Datacited by 1
  9. 2022Cloning Ideology and Style using Deep Learningcited by 0
  10. 2023A Semantic Approach to Negation Detection and Word Disambiguation with Natural Language Processingcited by 0
  11. 2023Time to Embrace Natural Language Processing (NLP)-based Digital Pathology: Benchmarking NLP- and Convolutional Neural Network-based Deep Learning Pipelinescited by 0
  12. 2023FlairNLP at SemEval-2023 Task 6b: Extraction of Legal Named Entities from Legal Texts using Contextual String Embeddingscited by 0
  13. 2023Empowering NLG: Offline Reinforcement Learning for Informal Summarization in Online Domainscited by 0
  14. 2023Prototype of a robotic system to assist the learning process of English language with text-generation through DNNcited by 0
  15. 2026DARTH-PUM: A Hybrid Processing-Using-Memory Architecturecited by 0