mlsanityData to 14 Oct 2026

Efficient Architecture Search via Bi-level Data Pruning

arXiv 2312.14200 ↗First posted 21 Dec 2023cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Improving the efficiency of Neural Architecture Search (NAS) is a challenging but significant task that has received much attention. Previous works mainly adopted the Differentiable Architecture Search (DARTS) and improved its search strategies or modules to enhance search efficiency. Recently, some methods have started considering data reduction for speedup, but they are not tightly coupled with the architecture search process, resulting in sub-optimal performance. To this end, this work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization, and then proposes a novel Bi-level Data Pruning (BDP) paradigm that targets the weights and architecture levels of DARTS to enhance efficiency from a data perspective. Specifically, we introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric, to gradually prune unsuitable samples for DARTS during the search. An effective automatic class balance constraint is also integrated into BDP, to suppress potential class imbalances resulting from data-efficient algorithms. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50% while achieving superior performance when applied to baseline DARTS. Besides, we demonstrate that BDP can harmoniously integrate with advanced DARTS variants, like PC-DARTS and beta-DARTS, offering an approximately 2 times speedup with minimal performance compromises.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 54 of its 75 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

54 of 75 references.

References · 54

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  4. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  8. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  12. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
38 more
  1. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  2. 2018SNAS: Stochastic Neural Architecture Searchcited by 289
  3. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  4. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  5. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  6. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  7. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  8. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 178
  9. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 131
  10. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  11. 2019MixConv: Mixed Depthwise Convolutional Kernelscited by 108
  12. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  13. 2018Searching for Efficient Multi-Scale Architectures for Dense Image Predictioncited by 96
  14. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  15. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  16. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  17. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  18. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 64
  19. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  20. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  21. 2019SpineNet: Learning Scale-Permuted Backbone for Recognition and Localizationcited by 50
  22. 2020ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Codingcited by 36
  23. 2018BlockQNN: Efficient Block-wise Neural Network Architecture Generationcited by 31
  24. 2020DOTS: Decoupling Operation and Topology in Differentiable Architecture Searchcited by 23
  25. 2021Core-set Sampling for Efficient Neural Architecture Searchcited by 23
  26. 2022Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairnesscited by 21
  27. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  28. 2020Cyclic Differentiable Architecture Searchcited by 18
  29. 2022Efficient Architecture Search for Diverse Taskscited by 18
  30. 2022BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruningcited by 17
  31. 2020Noisy Differentiable Architecture Searchcited by 16
  32. 2021Accelerating Neural Architecture Search via Proxy Datacited by 14
  33. 2022Efficient Joint-Dimensional Search with Solution Space Regularization for Real-Time Semantic Segmentationcited by 9
  34. 2019MoGA: Searching Beyond MobileNetV3cited by 8
  35. 2022Multi-scale Attentive Image De-raining Networks via Neural Architecture Searchcited by 8
  36. 2023β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Searchcited by 7
  37. 2022Speeding up NAS with Adaptive Subset Selectioncited by 6
  38. 2022Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Searchcited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  2. 2025Evolutionary Neural Architecture Search with Dual Contrastive Learningcited by 1