mlsanityData to 14 Oct 2026

Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search

arXiv 2307.15621 ↗First posted 28 Jul 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In this work, we show that simultaneously training and mixing neural networks is a promising way to conduct Neural Architecture Search (NAS). For hyperparameter optimization, reusing the partially trained weights allows for efficient search, as was previously demonstrated by the Population Based Training (PBT) algorithm. We propose PBT-NAS, an adaptation of PBT to NAS where architectures are improved during training by replacing poorly-performing networks in a population with the result of mixing well-performing ones and inheriting the weights using the shrink-perturb technique. After PBT-NAS terminates, the created networks can be directly used without retraining. PBT-NAS is highly parallelizable and effective: on challenging tasks (image generation and reinforcement learning) PBT-NAS achieves superior performance compared to baselines (random search and mutation-based PBT).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

41 of 55 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2018DeepMind Control Suitecited by 956
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  11. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  12. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
29 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  2. 2017Population Based Training of Neural Networkscited by 367
  3. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  4. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  5. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  6. 2020Neural Architecture Search without Trainingcited by 244
  7. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  8. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  9. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  10. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 153
  11. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  12. 2019NAS evaluation is frustratingly hardcited by 70
  13. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  14. 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 66
  15. 2016Network Morphismcited by 65
  16. 2020NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processingcited by 60
  17. 2016CMA-ES for Hyperparameter Optimization of Deep Neural Networkscited by 48
  18. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
  19. 2019Progressive DARTS: Bridging the Optimization Gap for NAS in the Wildcited by 40
  20. 2019AdversarialNAS: Adversarial Neural Architecture Search for GANscited by 38
  21. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 32
  22. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26
  23. 2021Core-set Sampling for Efficient Neural Architecture Searchcited by 23
  24. 2022Bayesian Generational Population-Based Trainingcited by 20
  25. 2021Faster Improvement Rate Population Based Trainingcited by 13
  26. 2022When Does Re-initialization Work?cited by 12
  27. 2020Safe Crossover of Neural Networks Through Neuron Alignmentcited by 8
  28. 2020Regularized Evolutionary Population-Based Trainingcited by 4
  29. 2020AlphaNet: An Attention Guided Deep Network for Automatic Image Mattingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Iterated Population Based Training with Task-Agnostic Restartscited by 0
  2. 2026Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networkscited by 0