mlsanityData to 7 Oct 2026

Navigating Scaling Laws: Compute Optimality in Adaptive Model Training

arXiv 2311.03233 ↗First posted 6 Nov 2023cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. The paradigm is very simple: investing more computational resources (optimally) leads to better performance, and even predictably so; neural scaling laws have been derived that accurately forecast the performance of a network for a desired level of compute. This leads to the notion of a `compute-optimal' model, i.e. a model that allocates a given level of compute during training optimally to maximize performance. In this work, we extend the concept of optimality by allowing for an `adaptive' model, i.e. a model that can change its shape during training. By doing so, we can design adaptive models that optimally traverse between the underlying scaling laws and outpace their `static' counterparts, leading to a significant reduction in the required compute to reach a given target performance. We show that our approach generalizes across modalities and different shape parameters.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 59 of its 70 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

59 of 70 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2021Scaling Vision Transformerscited by 993
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2021ImageNet-21K Pretraining for the Massescited by 607
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  14. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  15. 2018Born Again Neural Networkscited by 539
  16. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  17. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  18. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  19. 2017Population Based Training of Neural Networkscited by 351
  20. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  21. 2021Scaling Laws for Transfercited by 265
  22. 2021Curriculum Learning: A Surveycited by 257
  23. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  24. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  25. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  26. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  27. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  28. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  29. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  30. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  31. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  32. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  33. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  34. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  35. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  36. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  37. 2022Staged Training for Transformer Language Modelscited by 49
  38. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  2. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  3. 2026Efficient Pre-Training with Token Superpositioncited by 1
  4. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0