mlsanityData to 14 Oct 2026

Improving Transformer Models by Reordering their Sublayers

arXiv 1911.03864 ↗First posted 10 Nov 2019cs.CL cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We generate randomly ordered transformers and train them with the language modeling objective. We observe that some of these models are able to achieve better performance than the interleaved baseline, and that those successful variants tend to have more self-attention at the bottom and more feedforward sublayers at the top. We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time. However, the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models. Instead, we suggest that further exploration of task-specific sublayer reorderings is needed in order to unlock additional gains.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 10; 2023: 11; 2024: 8; 2025: 4; 2026: 3.

52 papers.

Sort:

By area:

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2021A Survey of Transformerscited by 401
  3. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  4. 2021Thinking Like Transformerscited by 184
  5. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  6. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  7. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  8. 2021Finetuning Pretrained Transformers into RNNscited by 96
  9. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  10. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  11. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  12. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
38 more
  1. 2023Energy Transformercited by 71
  2. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  3. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  6. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  7. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  8. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  9. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  10. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  11. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 21
  12. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  13. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16
  14. 2025SEDM: Selective Self-Evolving Distributed Memory for Agentscited by 16
  15. 2021Inspecting the concept knowledge graph encoded by modern language modelscited by 15
  16. 2021Staircase Attention for Recurrent Processing of Sequencescited by 14
  17. 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
  18. 2022A Neural ODE Interpretation of Transformer Layerscited by 14
  19. 2021RankNAS: Efficient Neural Architecture Search by Pairwise Rankingcited by 12
  20. 2021Revisiting Simple Neural Probabilistic Language Modelscited by 10
  21. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  22. 2023Breaking through the learning plateaus of in-context learning in Transformercited by 8
  23. 2023Centered Self-Attention Layerscited by 7
  24. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  25. 2021IOT: Instance-wise Layer Reordering for Transformer Structurescited by 5
  26. 2025Improving Routing in Sparse Mixture of Experts with Graph of Tokenscited by 4
  27. 2026Distinct mechanisms underlying in-context learning in transformerscited by 4
  28. 2022TrimBERT: Tailoring BERT for Trade-offscited by 3
  29. 2023NarrowBERT: Accelerating Masked Language Model Pretraining and Inferencecited by 3
  30. 2023Interpretability in Activation Space Analysis of Transformers: A Focused Surveycited by 3
  31. 2023Evolutionary Neural Architecture Search for Transformer in Knowledge Tracingcited by 3
  32. 2024Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillationcited by 3
  33. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  34. 2022FL-Tuning: Layer Tuning for Feed-Forward Network in Transformercited by 2
  35. 2024Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Promptscited by 2
  36. 2022Efficient Sparsely Activated Transformerscited by 1
  37. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1
  38. 2026An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammarscited by 1

50 most cited of 52 All 52 (JSON)