mlsanityData to 7 Oct 2026

A Survey of Transformers

arXiv 2106.04554 ↗First posted 8 Jun 2021cs.LG cs.AI cs.CLcited by 399

Not on the map. Why?

Abstract (arXiv)

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

Key facts. Cited by 399 papers in the map (data through 7 Oct 2026). Builds on 144 of its 180 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

144 of 180 references.

References · 144

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  13. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  14. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  15. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  16. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  17. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  18. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  19. 2020GLU Variants Improve Transformercited by 1,724
  20. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  21. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  22. 2020Reformer: The Efficient Transformercited by 1,658
  23. 2021ViViT: A Video Vision Transformercited by 1,610
  24. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  25. 2020Rethinking Attention with Performerscited by 1,584
  26. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  27. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  28. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  29. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  30. 2018Self-Attention with Relative Position Representationscited by 1,112
  31. 2021Transformers in Vision: A Surveycited by 1,009
  32. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  33. 2020On Layer Normalization in the Transformer Architecturecited by 906
  34. 2018Image Transformercited by 770
  35. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  36. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  37. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  38. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691

50 most cited of 144 All 144 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 58; 2023: 118; 2024: 121; 2025: 65; 2026: 19.

399 papers.

Sort:

By area:

  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2022Transformers in Medical Imaging: A Surveycited by 245
  5. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  7. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  8. 2023A mathematical perspective on Transformerscited by 160
  9. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  10. 2023The emergence of clusters in self-attention dynamicscited by 133
  11. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
38 more
  1. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  2. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 118
  3. 2021A Survey of Visual Transformerscited by 117
  4. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  5. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
  6. 2023Attending to Graph Transformerscited by 81
  7. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  8. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  9. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
  10. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  11. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  12. 2022Formal Algorithms for Transformerscited by 70
  13. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  14. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  15. 2023Energy Transformercited by 69
  16. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  17. 2022Video Transformers: A Surveycited by 63
  18. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  19. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  20. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  21. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  22. 2022EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognitioncited by 48
  23. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  24. 2022Error Correction Code Transformercited by 47
  25. 2023A Survey on Long Text Modeling with Transformerscited by 47
  26. 2023PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformercited by 46
  27. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  28. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  29. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  30. 2024Toto: Time Series Optimized Transformer for Observabilitycited by 44
  31. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  32. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  33. 2022TACTiS: Transformer-Attentional Copulas for Time Seriescited by 38
  34. 2023Transformers in Speech Processing: A Surveycited by 38
  35. 2024Large Language Models for Medicine: A Surveycited by 38
  36. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  37. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  38. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35

50 most cited of 399 All 399 (JSON)