mlsanityData to 14 Oct 2026

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

arXiv 2207.10551 ↗First posted 21 Jul 2022cs.LG cs.CLcited by 103

Not on the map. Why?

Abstract (arXiv)

There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 21 Jul 2022.

Builds on

37 of 45 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
25 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2020Efficient Transformers: A Surveycited by 790
  8. 2016Language Modeling with Gated Convolutional Networkscited by 702
  9. 2018Universal Transformerscited by 647
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  11. 2021Explaining Neural Scaling Lawscited by 363
  12. 2022UL2: Unifying Language Learning Paradigmscited by 295
  13. 2021Scaling Laws for Transfercited by 272
  14. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  15. 2019The Evolved Transformercited by 180
  16. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  17. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  18. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  19. 2021The Benchmark Lotterycited by 136
  20. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  21. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  22. 2021The Efficiency Misnomercited by 98
  23. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  24. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  25. 2020Transferring Inductive Biases through Knowledge Distillationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 27; 2024: 32; 2025: 18; 2026: 10.

103 papers.

Sort:

By area:

  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  3. 2023BloombergGPT: A Large Language Model for Financecited by 939
  4. 2022Galactica: A Large Language Model for Sciencecited by 744
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  6. 2023Scaling Data-Constrained Language Modelscited by 403
  7. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  8. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  9. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  10. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  12. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
38 more
  1. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  2. 2024Scaling Laws for Precisioncited by 95
  3. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  4. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  5. 2022Training Trajectories of Language Models Across Scalescited by 81
  6. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  7. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  8. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  9. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  10. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  11. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
  12. 2025Horizon Reduction Makes RL Scalablecited by 57
  13. 2024Base of RoPE Bounds Context Lengthcited by 52
  14. 2024Does equivariance matter at scale?cited by 52
  15. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  16. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  17. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  18. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  19. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  20. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  21. 2023Sparse Universal Transformercited by 32
  22. 2024Algorithmic progress in language modelscited by 31
  23. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  24. 2022What Do NLP Researchers Believe? Results of the NLP Community Metasurveycited by 28
  25. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  26. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  27. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  28. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  29. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  30. 2024TransformerFAM: Feedback attention is working memorycited by 22
  31. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 22
  32. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  33. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
  34. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  35. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  36. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  37. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  38. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15

50 most cited of 103 All 103 (JSON)