mlsanityData to 13 Oct 2026

Entropic Score metric: Decoupling Topology and Size in Training-free NAS

arXiv 2310.04179 ↗First posted 6 Oct 2023cs.CV cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

51 of 59 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,786
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  11. 2022A ConvNet for the 2020scited by 3,563
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,544
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,789
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,496
  7. 2017Random Erasing Data Augmentationcited by 1,238
  8. 2016Deep Networks with Stochastic Depthcited by 977
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 907
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 809
  11. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 639
  12. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  13. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 476
  14. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  15. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 469
  16. 2017Large-Scale Evolution of Image Classifierscited by 348
  17. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 341
  18. 2021XCiT: Cross-Covariance Image Transformerscited by 337
  19. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 259
  20. 2020Neural Architecture Search without Trainingcited by 242
  21. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  22. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  23. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  24. 2021Zero-Cost Proxies for Lightweight NAScited by 196
  25. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  26. 2016Deep Pyramidal Residual Networkscited by 163
  27. 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
  28. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  29. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  30. 2022Inception Transformercited by 99
  31. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  32. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 72
  33. 2020Talking-Heads Attentioncited by 58
  34. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  35. 2021Searching the Search Space of Vision Transformercited by 40
  36. 2022Training-free Transformer Architecture Searchcited by 33
  37. 2021ViTAS: Vision Transformer Architecture Searchcited by 31
  38. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variationcited by 0
  2. 2025Half Search Space is All You Needcited by 0