mlsanityData to 7 Oct 2026

Do Transformer Modifications Transfer Across Implementations and Applications?

arXiv 2102.11972 ↗First posted 23 Feb 2021cs.LG cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

59 of 74 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Layer Normalizationcited by 5,579
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  13. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  14. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  15. 2020On Layer Normalization in the Transformer Architecturecited by 906
  16. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  17. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  18. 2017Deep Reinforcement Learning that Matterscited by 727
  19. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  20. 2016Language Modeling with Gated Convolutional Networkscited by 688
  21. 2018Universal Transformerscited by 613
  22. 2013Network In Networkcited by 600
  23. 2017Self-Normalizing Neural Networkscited by 591
  24. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  25. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  26. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  27. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  28. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  29. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  30. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  31. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  32. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  33. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  34. 2019The Evolved Transformercited by 180
  35. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  36. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  37. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  38. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 28; 2023: 14; 2024: 12; 2025: 4; 2026: 9.

87 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  8. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  9. 2022UL2: Unifying Language Learning Paradigmscited by 292
  10. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2022Transformer Quality in Linear Timecited by 189
38 more
  1. 2023Language-Driven Representation Learning for Roboticscited by 163
  2. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  3. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  4. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  5. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  6. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  7. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  8. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  9. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  10. 2021The Efficiency Misnomercited by 98
  11. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  12. 2021Sparse is Enough in Scaling Transformerscited by 86
  13. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  14. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  15. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  16. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  17. 2021Distributed Deep Learning in Open Collaborationscited by 58
  18. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  19. 2022Pretraining Without Attentioncited by 51
  20. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  21. 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
  22. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  23. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  24. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  25. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  26. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  27. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  28. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  29. 2022What Do NLP Researchers Believe? Results of the NLP Community Metasurveycited by 28
  30. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  31. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  32. 2024Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMscited by 20
  33. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  34. 2022Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrievalcited by 19
  35. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  36. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  37. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12
  38. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12

50 most cited of 87 All 87 (JSON)