mlsanityData to 14 Oct 2026

A Study on Transformer Configuration and Training Objective

arXiv 2205.10505 ↗First posted 21 May 2022cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Transformer-based models have delivered impressive results on many tasks, particularly vision and language tasks. In many model training situations, conventional configurations are typically adopted. For example, we often set the base model with hidden dimensions (i.e. model width) to be 768 and the number of transformer layers (i.e. model depth) to be 12. In this paper, we revisit these conventional configurations. Through theoretical analysis and experimental evaluation, we show that the masked autoencoder is effective in alleviating the over-smoothing issue in deep transformer training. Based on this finding, we propose Bamboo, an idea of using deeper and narrower transformer configurations, for masked autoencoder training. On ImageNet, with such a simple change in configuration, re-designed model achieves 87.1% top-1 accuracy and outperforms SoTA models like MAE and BEiT. On language tasks, re-designed model outperforms BERT with default setting by 1.1 points on average, on GLUE datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

36 of 43 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
24 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2016Deep Networks with Stochastic Depthcited by 979
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  8. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  10. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  11. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  12. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  13. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  14. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  15. 2019On Identifiability in Transformerscited by 116
  16. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  17. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  18. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  19. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  20. 2021Go Wider Instead of Deepercited by 57
  21. 2021Vision Transformers with Patch Diversificationcited by 45
  22. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  23. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  24. 2020An Embarrassingly Simple Model for Dialogue Relation Extractioncited by 13

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024Transformer Layers as Painterscited by 54
  2. 2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuningcited by 2
  3. 2023Polynomial-based Self-Attention for Table Representation learningcited by 1
  4. 2024Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformerscited by 1