mlsanityData to 7 Oct 2026

Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift

arXiv 2108.09671 ↗First posted 22 Aug 2021cs.CV cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Recently proposed neural architecture search (NAS) methods co-train billions of architectures in a supernet and estimate their potential accuracy using the network weights detached from the supernet. However, the ranking correlation between the architectures' predicted accuracy and their actual capability is incorrect, which causes the existing NAS methods' dilemma. We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift. Feature shift is identified as dynamic input distributions of a hidden layer due to random path sampling. The input distribution dynamic affects the loss descent and finally affects architecture ranking. Parameter shift is identified as contradictory parameter updates for a shared layer lay in different paths in different training steps. The rapidly-changing parameter could not preserve architecture ranking. We address these two shifts simultaneously using a nontrivial supernet-Pi model, called Pi-NAS. Specifically, we employ a supernet-Pi model that contains cross-path learning to reduce the feature consistency shift between different paths. Meanwhile, we adopt a novel nontrivial mean teacher containing negative samples to overcome parameter shift and model collision. Furthermore, our Pi-NAS runs in an unsupervised manner, which can search for more transferable architectures. Extensive experiments on ImageNet and a wide range of downstream tasks (e.g., COCO 2017, ADE20K, and Cityscapes) demonstrate the effectiveness and universality of our Pi-NAS compared to supervised NAS. See Codes: https://github.com/Ernie1/Pi-NAS.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 22 Aug 2021.

Builds on

52 of 67 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Mask R-CNNcited by 7,259
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
38 more
  1. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  2. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  7. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  8. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  9. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  11. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  12. 2020ResNeSt: Split-Attention Networkscited by 445
  13. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  14. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  15. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  16. 2019Selective Kernel Networkscited by 401
  17. 2017Large-Scale Evolution of Image Classifierscited by 348
  18. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  19. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  20. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  21. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  22. 2014Learning with Pseudo-Ensemblescited by 241
  23. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  24. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  25. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  26. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
  27. 2021Dynamic Slimmable Networkcited by 83
  28. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  29. 2020Can weight sharing outperform random architecture search? An investigation with TuNAScited by 68
  30. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  31. 2020Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?cited by 58
  32. 2021Solving Inefficiency of Self-supervised Representation Learningcited by 41
  33. 2020Are Labels Necessary for Neural Architecture Search?cited by 37
  34. 2021Neural Architecture Search with Random Labelscited by 33
  35. 2017DeepArchitect: Automatically Designing and Training Deep Architecturescited by 28
  36. 2019Improving One-shot NAS by Suppressing the Posterior Fadingcited by 28
  37. 2019Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Searchcited by 23
  38. 2019Sample-Efficient Neural Architecture Search by Learning Action Spacecited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 3; 2025: 5; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  2. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Accelerationcited by 70
  3. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  4. 2024FrugalNeRF: Fast Convergence for Extreme Few-shot Novel View Synthesis without Learned Priorscited by 18
  5. 2022NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimensioncited by 5
  6. 2024DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisionscited by 4
  7. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 4
  8. 2025Subnet-Aware Dynamic Supernet Training for Neural Architecture Searchcited by 4
  9. 2025An Encode-then-Decompose Approach to Unsupervised Time Series Anomaly Detection on Contaminated Training Data--Extended Versioncited by 3
  10. 2022Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAScited by 1
  11. 2025DDS-NAS: Dynamic Data Selection within Neural Architecture Search via On-line Hard Example Mining applied to Image Classificationcited by 1
  12. 2025Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learningcited by 1
2 more
  1. 2025Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Modelscited by 0
  2. 2026TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Expertscited by 0