mlsanityData to 7 Oct 2026

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

arXiv 2211.12759 ↗First posted 23 Nov 2022cs.CV cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

One-shot neural architecture search (NAS) substantially improves the search efficiency by training one supernet to estimate the performance of every possible child architecture (i.e., subnet). However, the inconsistency of characteristics among subnets incurs serious interference in the optimization, resulting in poor performance ranking correlation of subnets. Subsequent explorations decompose supernet weights via a particular criterion, e.g., gradient matching, to reduce the interference; yet they suffer from huge computational cost and low space separability. In this work, we propose a lightweight and effective local intrinsic dimension (LID)-based method NAS-LID. NAS-LID evaluates the geometrical properties of architectures by calculating the low-cost LID features layer-by-layer, and the similarity characterized by LID enjoys better separability compared with gradients, which thus effectively reduces the interference among subnets. Extensive experiments on NASBench-201 indicate that NAS-LID achieves superior performance with better efficiency. Specifically, compared to the gradient-driven method, NAS-LID can save up to 86% of GPU memory overhead when searching on NASBench-201. We also demonstrate the effectiveness of NAS-LID on ProxylessNAS and OFA spaces. Source code: https://github.com/marsggbo/NAS-LID.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

46 of 54 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  8. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  9. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  10. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  11. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
34 more
  1. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  4. 2017Progressive Neural Architecture Searchcited by 483
  5. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  6. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  7. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  9. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  10. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  11. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  12. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  13. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  14. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  15. 2020DrNAS: Dirichlet Neural Architecture Searchcited by 73
  16. 2019One-Shot Neural Architecture Search via Self-Evaluated Template Networkcited by 69
  17. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  18. 2019BayesNAS: A Bayesian Approach for Neural Architecture Searchcited by 67
  19. 2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61
  20. 2020Few-shot Neural Architecture Searchcited by 50
  21. 2018Geometric Understanding of Deep Learningcited by 41
  22. 2019Progressive DARTS: Bridging the Optimization Gap for NAS in the Wildcited by 39
  23. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  24. 2020ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Codingcited by 36
  25. 2019Search to Distill: Pearls are Everywhere but not the Eyescited by 26
  26. 2022Generalizing Few-Shot NAS with Gradient Matchingcited by 25
  27. 2021K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernetscited by 22
  28. 2021Analyzing and Mitigating Interference in Neural Architecture Searchcited by 20
  29. 2021Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shiftcited by 14
  30. 2021Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scanscited by 13
  31. 2018Teacher Guided Architecture Searchcited by 10
  32. 2021Evolutionary Multi-objective Architecture Search Framework: Application to COVID-19 3D CT Classificationcited by 5
  33. 2021EAGAN: Efficient Two-stage Evolutionary Architecture Search for GANscited by 5
  34. 2022CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAScited by 5

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  2. 2024CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Frameworkcited by 5
  3. 2025Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contributioncited by 2
  4. 2024Multi-Objective Neural Architecture Search by Learning Search Space Partitionscited by 1
  5. 2025Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Storycited by 1