mlsanityData to 7 Oct 2026

Mutually-aware Sub-Graphs Differentiable Architecture Search

arXiv 2107.04324 ↗First posted 9 Jul 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Differentiable architecture search is prevalent in the field of NAS because of its simplicity and efficiency, where two paradigms, multi-path algorithms and single-path methods, are dominated. Multi-path framework (e.g. DARTS) is intuitive but suffers from memory usage and training collapse. Single-path methods (e.g.GDAS and ProxylessNAS) mitigate the memory issue and shrink the gap between searching and evaluation but sacrifice the performance. In this paper, we propose a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS). The core of our framework is a differentiable Gumbel-TopK sampler that produces multiple mutually exclusive single-path sub-graphs. To alleviate the severer skip-connect issue brought by multiple sub-graphs setting, we propose a Dropblock-Identity module to stabilize the optimization. To make best use of the available models (super-net and sub-graphs), we introduce a memory-efficient super-net guidance distillation to improve training. The proposed framework strikes a balance between flexible memory usage and searching quality. We demonstrate the effectiveness of our methods on ImageNet and CIFAR10, where the searched models show a comparable performance as the most recent approaches.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 57 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

49 of 57 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
37 more
  1. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  4. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  10. 2017Large-Scale Evolution of Image Classifierscited by 348
  11. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  12. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  13. 2018DropBlock: A regularization method for convolutional networkscited by 261
  14. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  15. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  16. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  17. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  18. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  19. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  20. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  21. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  22. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
  23. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 129
  24. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  25. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  26. 2018Neural Nearest Neighbors Networkscited by 98
  27. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  28. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  29. 2019NAS evaluation is frustratingly hardcited by 69
  30. 2019NAS-FCOS: Fast Neural Architecture Search for Object Detectioncited by 53
  31. 2018Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cellscited by 39
  32. 2021Neural Architecture Search with Random Labelscited by 33
  33. 2019Search to Distill: Pearls are Everywhere but not the Eyescited by 26
  34. 2020Cyclic Differentiable Architecture Searchcited by 18
  35. 2020Representation Sharing for Fast Object Detector Search and Beyondcited by 9
  36. 2019Differentiable Architecture Search with Ensemble Gumbel-Softmaxcited by 6
  37. 2021Unchain the Search Space with Hierarchical Differentiable Architecture Searchcited by 1

Built on by

Not cited yet.