mlsanityData to 14 Oct 2026

Foundation Transformers

arXiv 2210.06423 ↗First posted 12 Oct 2022cs.LG cs.CL cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name "Transformers", the above areas use different implementations for better performance, e.g., Post-LayerNorm for BERT, and Pre-LayerNorm for GPT and vision Transformers. We call for the development of Foundation Transformer for true general-purpose modeling, which serves as a go-to architecture for various tasks and modalities with guaranteed training stability. In this work, we introduce a Transformer variant, named Magneto, to fulfill the goal. Specifically, we propose Sub-LayerNorm for good expressivity, and the initialization strategy theoretically derived from DeepNet for stable scaling up. Extensive experiments demonstrate its superior performance and better stability than the de facto Transformer variants designed for various applications, including language modeling (i.e., BERT, and GPT), machine translation, vision pretraining (i.e., BEiT), speech recognition, and multimodal pretraining (i.e., BEiT-3).

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
20 more
  1. 2019Natural Adversarial Examplescited by 1,408
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  10. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  13. 2020Understanding the Difficulty of Training Transformerscited by 210
  14. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  15. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  16. 2022Language Models are General-Purpose Interfacescited by 97
  17. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  18. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  19. 2019SpecAugment on Large Scale Datasetscited by 60
  20. 2022A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategiescited by 15

Built on by

Cited by year

Citing papers per year: 2023: 12; 2024: 5; 2025: 2.

19 papers.

Sort:

By area:

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  3. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  4. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  6. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  7. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  9. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  10. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  11. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  12. 2024EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learningcited by 25
7 more
  1. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  2. 2023Dual PatchNormcited by 9
  3. 2024SeqNAS: Neural Architecture Search for Event Sequence Classificationcited by 8
  4. 2023RevColV2: Exploring Disentangled Representations in Masked Image Modelingcited by 4
  5. 2024MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Descriptioncited by 3
  6. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  7. 2024Enhancing Vision-Language Model with Unmasked Token Alignmentcited by 0