mlsanityData to 14 Oct 2026

SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS

arXiv 2403.04161 ↗First posted 7 Mar 2024cs.LG cs.CV cs.NEcited by 11

Not on the map. Why?

Abstract (arXiv)

Training-free metrics (a.k.a. zero-cost proxies) are widely used to avoid resource-intensive neural network training, especially in Neural Architecture Search (NAS). Recent studies show that existing training-free metrics have several limitations, such as limited correlation and poor generalisation across different search spaces and tasks. Hence, we propose Sample-Wise Activation Patterns and its derivative, SWAP-Score, a novel high-performance training-free metric. It measures the expressivity of networks over a batch of input samples. The SWAP-Score is strongly correlated with ground-truth performance across various search spaces and tasks, outperforming 15 existing training-free metrics on NAS-Bench-101/201/301 and TransNAS-Bench-101. The SWAP-Score can be further enhanced by regularisation, which leads to even higher correlations in cell-based search space and enables model size control during the search. For example, Spearman's rank correlation coefficient between regularised SWAP-Score and CIFAR-100 validation accuracies on NAS-Bench-201 networks is 0.90, significantly higher than 0.80 from the second-best metric, NWOT. When integrated with an evolutionary algorithm for NAS, our SWAP-NAS achieves competitive performance on CIFAR-10 and ImageNet in approximately 6 minutes and 9 minutes of GPU time respectively.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

50 of 61 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  11. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  12. 2017Progressive Neural Architecture Searchcited by 487
38 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  3. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  5. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  6. 2017Large-Scale Evolution of Image Classifierscited by 348
  7. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  8. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  9. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  10. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
  11. 2020Neural Architecture Search without Trainingcited by 244
  12. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  13. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  14. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  15. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  16. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 131
  17. 2019Neural Predictor for Neural Architecture Searchcited by 118
  18. 2020BRP-NAS: Prediction-based NAS using GCNscited by 111
  19. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  20. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  21. 2019CARS: Continuous Evolution for Efficient Neural Architecture Searchcited by 79
  22. 2020NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Searchcited by 76
  23. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 73
  24. 2019One-Shot Neural Architecture Search via Self-Evaluated Template Networkcited by 69
  25. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  26. 2021TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Searchcited by 58
  27. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  28. 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 55
  29. 2020Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gapcited by 50
  30. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  31. 2020Semi-Supervised Neural Architecture Searchcited by 46
  32. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  33. 2021Contrastive Neural Architecture Search with Neural Architecture Comparatorscited by 34
  34. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26
  35. 2021EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Searchcited by 24
  36. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  37. 2019EENA: Efficient Evolution of Neural Architecturecited by 10
  38. 2020Evolving Neural Architecture Using One Shot Modelcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 5.

11 papers.

  1. 2025SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Modelscited by 7
  2. 2025L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformerscited by 5
  3. 2026Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffscited by 3
  4. 2024Zero-Shot NAS via the Suppression of Local Entropy Decreasecited by 2
  5. 2024NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performancecited by 1
  6. 2025Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvaturecited by 1
  7. 2026CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Searchcited by 1
  8. 2025Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variationcited by 0
  9. 2026XTinyU-Net: Training-Free U-Net Scaling via Initialization-Time Sensitivitycited by 0
  10. 2026Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discoverycited by 0
  11. 2026RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Searchcited by 0