# Generalizing Few-Shot NAS with Gradient Matching

arXiv: https://arxiv.org/abs/2203.15207 · HTML: https://mlsanity.com/paper/2203.15207/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 62 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)

## Built on by (most cited first, 25 of 25)

- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [Efficient Non-Parametric Optimizer Search for Diverse Tasks](https://mlsanity.com/paper/2209.13575/) (2022)
- [PA&DA: Jointly Sampling PAth and DAta for Consistent NAS](https://mlsanity.com/paper/2302.14772/) (2023)
- [DC-CCL: Device-Cloud Collaborative Controlled Learning for Large Vision Models](https://mlsanity.com/paper/2303.10361/) (2023)
- [NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension](https://mlsanity.com/paper/2211.12759/) (2022)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach](https://mlsanity.com/paper/2403.11380/) (2024)
- [One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model](https://mlsanity.com/paper/2406.10160/) (2024)
- [Subnet-Aware Dynamic Supernet Training for Neural Architecture Search](https://mlsanity.com/paper/2503.10740/) (2025)
- [Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning](https://mlsanity.com/paper/2505.05226/) (2025)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)
- [Forward Once for All: Structural Parameterized Adaptation for Efficient Cloud-coordinated On-device Recommendation](https://mlsanity.com/paper/2501.02837/) (2025)
- [A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search](https://mlsanity.com/paper/2204.03916/) (2022)
- [Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution](https://mlsanity.com/paper/2506.01231/) (2025)
- [CARL: Causality-guided Architecture Representation Learning for an Interpretable Performance Predictor](https://mlsanity.com/paper/2506.04001/) (2025)
- [Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAS](https://mlsanity.com/paper/2206.05896/) (2022)
- [Balanced Mixture of SuperNets for Learning the CNN Pooling Architecture](https://mlsanity.com/paper/2306.11982/) (2023)
- [Learning to Compose SuperWeights for Neural Parameter Allocation Search](https://mlsanity.com/paper/2312.01274/) (2023)
- [Multi-Objective Neural Architecture Search by Learning Search Space Partitions](https://mlsanity.com/paper/2406.00291/) (2024)
- [DBsurf: A Discrepancy Based Method for Discrete Stochastic Gradient Estimation](https://mlsanity.com/paper/2309.03974/) (2023)
- [HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning](https://mlsanity.com/paper/2412.10723/) (2024)
- [Neural Architecture Search by Learning a Hierarchical Search Space](https://mlsanity.com/paper/2503.21061/) (2025)
- [InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search](https://mlsanity.com/paper/2606.18676/) (2026)
- [LESS: Lightweight Evolutionary Supernet Search in Minutes](https://mlsanity.com/paper/2610.01468/) (2026)

## Abstract (arXiv)

Efficient performance estimation of architectures drawn from large search spaces is essential to Neural Architecture Search. One-Shot methods tackle this challenge by training one supernet to approximate the performance of every architecture in the search space via weight-sharing, thereby drastically reducing the search cost. However, due to coupled optimization between child architectures caused by weight-sharing, One-Shot supernet's performance estimation could be inaccurate, leading to degraded search outcomes. To address this issue, Few-Shot NAS reduces the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets via edge-wise (layer-wise) exhaustive partitioning. Since each partition of the supernet is not equally important, it necessitates the design of a more effective splitting criterion. In this work, we propose a gradient matching score (GM) that leverages gradient information at the shared weight for making informed splitting decisions. Intuitively, gradients from different child models can be used to identify whether they agree on how to update the shared modules, and subsequently to decide if they should share the same weight. Compared with exhaustive partitioning, the proposed criterion significantly reduces the branching factor per edge. This allows us to split more edges (layers) for a given budget, resulting in substantially improved performance as NAS search spaces usually include dozens of edges (layers). Extensive empirical evaluations of the proposed method on a wide range of search spaces (NASBench-201, DARTS, MobileNet Space), datasets (cifar10, cifar100, ImageNet) and search algorithms (DARTS, SNAS, RSPS, ProxylessNAS, OFA) demonstrate that it significantly outperforms its Few-Shot counterparts while surpassing previous comparable methods in terms of the accuracy of derived architectures.