mlsanityData to 14 Oct 2026

Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design

arXiv 2305.13035 ↗First posted 22 May 2023cs.CV cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-G/14 under identical settings, with also less than half the inference cost. We conduct a thorough evaluation across multiple tasks, such as image classification, captioning, VQA and zero-shot transfer, demonstrating the effectiveness of our model across a broad range of domains and identifying limitations. Overall, our findings challenge the prevailing approach of blindly scaling up vision models and pave a path for a more informed scaling.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 66 of its 82 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

66 of 82 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020Language Models are Few-Shot Learnerscited by 32,419
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2016Wide Residual Networkscited by 3,070
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2021Convex optimizationcited by 2,515
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  13. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  15. 2019Natural Adversarial Examplescited by 1,408
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  18. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  19. 2021Scaling Vision Transformerscited by 999
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  21. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  22. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  23. 2018Panoptic Segmentationcited by 802
  24. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  26. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  27. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  28. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  29. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  30. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  31. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  32. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  33. 2022DeiT III: Revenge of the ViTcited by 419
  34. 2021Explaining Neural Scaling Lawscited by 363
  35. 2020Are we done with ImageNet?cited by 333
  36. 2021Scaling Laws for Transfercited by 272
  37. 2019Fixing the train-test resolution discrepancycited by 209
  38. 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 31; 2025: 38; 2026: 18.

89 papers.

Sort:

By area:

  1. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  3. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  4. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  5. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  6. 2025Your ViT is Secretly an Image Segmentation Modelcited by 63
  7. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  8. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  9. 2024Does equivariance matter at scale?cited by 52
  10. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  11. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 46
  12. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
38 more
  1. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  2. 2024Scaling Optimal LR Across Token Horizonscited by 40
  3. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  4. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  5. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  6. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 37
  7. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  8. 2024Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Contentcited by 22
  9. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  10. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Modelscited by 21
  11. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  12. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  13. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  14. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Bettercited by 17
  15. 2025Scaling Inference-Efficient Language Modelscited by 16
  16. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 16
  17. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  18. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 15
  19. 2024Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Predictioncited by 13
  20. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  21. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 13
  22. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  23. 2026C-RADIOv4 (Tech Report)cited by 12
  24. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?cited by 11
  25. 2026EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformerscited by 11
  26. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 10
  27. 2025Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systemscited by 10
  28. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMscited by 10
  29. 2024FlexCap: Describe Anything in Images in Controllable Detailcited by 7
  30. 2024Scaling Sequential Recommendation Models with Transformerscited by 7
  31. 2025Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Modelscited by 7
  32. 2025DOFA-CLIP: Multimodal Vision-Language Foundation Models for Earth Observationcited by 7
  33. 2025A Recipe for Improving Remote Sensing VLM Zero Shot Generalizationcited by 7
  34. 2024A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysiscited by 6
  35. 2026Scaling laws for amplitude surrogatescited by 6
  36. 2024Micrometer: Micromechanics Transformer for Predicting Mechanical Responses of Heterogeneous Materialscited by 5
  37. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  38. 2026Bi-Orthogonal Factor Decomposition for Vision Transformerscited by 5

50 most cited of 89 All 89 (JSON)