mlsanityData to 7 Oct 2026

ScaleNet: Searching for the Model to Scale

arXiv 2207.07267 ↗First posted 15 Jul 2022cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, community has paid increasing attention on model scaling and contributed to developing a model family with a wide spectrum of scales. Current methods either simply resort to a one-shot NAS manner to construct a non-structural and non-scalable model family or rely on a manual yet fixed scaling strategy to scale an unnecessarily best base model. In this paper, we bridge both two components and propose ScaleNet to jointly search base model and scaling strategy so that the scaled large model can have more promising performance. Concretely, we design a super-supernet to embody models with different spectrum of sizes (e.g., FLOPs). Then, the scaling strategy can be learned interactively with the base model via a Markov chain-based evolution algorithm and generalized to develop even larger models. To obtain a decent super-supernet, we design a hierarchical sampling strategy to enhance its training sufficiency and alleviate the disturbance. Experimental results show our scaled networks enjoy significant performance superiority on various FLOPs, but with at least 2.53x reduction on search cost. Codes are available at https://github.com/luminolx/ScaleNet.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

40 of 54 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2016Wide Residual Networkscited by 3,057
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2019Searching for MobileNetV3cited by 1,984
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
28 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  2. 2021Scaling Vision Transformerscited by 993
  3. 2020Designing Network Design Spacescited by 954
  4. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  5. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  6. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  7. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  8. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  9. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  10. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  11. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  12. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  13. 2020Neural Architecture Transfercited by 59
  14. 2020Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetscited by 49
  15. 2021Fast and Accurate Model Scalingcited by 46
  16. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  17. 2018Evolutionary Generative Adversarial Networkscited by 42
  18. 2021Prioritized Architecture Sampling with Monto-Carlo Tree Searchcited by 32
  19. 2018Domain Adaptive Transfer Learning with Specialist Modelscited by 31
  20. 2021ViTAS: Vision Transformer Architecture Searchcited by 31
  21. 2021Locally Free Weight Sharing for Network Width Searchcited by 27
  22. 2021BCNet: Searching for Network Width with Bilaterally Coupled Networkcited by 23
  23. 2021Searching for Fast Model Families on Datacenter Acceleratorscited by 22
  24. 2021Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platformscited by 18
  25. 2020Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimizationcited by 9
  26. 2020GPCA: A Probabilistic Framework for Gaussian Process Embedded Channel Attentioncited by 7
  27. 2020DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inferencecited by 7
  28. 2021Greedy Network Enlargingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 0; 2026: 1.

4 papers.

  1. 2023EQ-Net: Elastic Quantization Neural Networkscited by 10
  2. 2024Differentiable Model Scaling using Differentiable Topkcited by 2
  3. 2024CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloadscited by 1
  4. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0