mlsanityData to 7 Oct 2026

Transformer models: an introduction and catalog

arXiv 2302.07730 ↗First posted 12 Feb 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

In the past few years we have seen the meteoric appearance of dozens of foundation models of the Transformer family, all of which have memorable and sometimes funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models. The paper also includes an introduction to the most important aspects and innovations in Transformer models. Our catalog will include models that are trained using self-supervised learning (e.g., BERT or GPT3) as well as those that are further trained using a human-in-the-loop (e.g. the InstructGPT model used by ChatGPT).

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 70 of its 105 references that are in the map. First posted on arXiv 12 Feb 2023.

Builds on

70 of 105 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  13. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  14. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  15. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  16. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  17. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  18. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  19. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  20. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  21. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  22. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  23. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  24. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  25. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  26. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  27. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  28. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  29. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  30. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  31. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  32. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  33. 2021Transformers in Vision: A Surveycited by 1,009
  34. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  35. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  36. 2022A Generalist Agentcited by 802
  37. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  38. 2022Galactica: A Large Language Model for Sciencecited by 736

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 9; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Large Language Models: A Surveycited by 506
  2. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  3. 2023LLMs4OL: Large Language Models for Ontology Learningcited by 75
  4. 2024Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Spacecited by 39
  5. 2025LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Modelscited by 9
  6. 2023Segment Anything Meets Semantic Communicationcited by 8
  7. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  8. 2024Development of a Reliable and Accessible Caregiving Language Model (CaLM)cited by 4
  9. 2025Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Modelscited by 4
  10. 2025Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluationcited by 4
  11. 2025GreenAuto: An Automated Platform for Sustainable AI Model Design on Edge Devicescited by 3
  12. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
9 more
  1. 2025Diffusion Models for Hyperspectral Image Analysis: A Comprehensive Reviewcited by 2
  2. 2023When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dustcited by 1
  3. 2024FGATT: A Robust Framework for Wireless Data Imputation Using Fuzzy Graph Attention Networks and Transformer Encoderscited by 1
  4. 2025ExBigBang: A Dynamic Approach for Explainable Persona Classification through Contextualized Hybrid Transformer Analysiscited by 1
  5. 2026GAT-NeRF: Geometry-Aware-Transformer Enhanced Neural Radiance Fields for High-Fidelity 4D Facial Avatarscited by 1
  6. 2025AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customizationcited by 0
  7. 2025A Hybrid Framework for Subject Analysis: Integrating Embedding-Based Regression Models with Large Language Modelscited by 0
  8. 2025SwiftEmbed: Ultra-Fast Text Embeddings via Static Token Lookup for Real-Time Applicationscited by 0
  9. 2026IG-GAN: A Generative Adversarial Network for Aerodynamic Data Generation Based on Intrinsic Geometrycited by 0