mlsanityData to 14 Oct 2026

Exploring the Intersection between Neural Architecture Search and Continual Learning

arXiv 2206.05625 ↗First posted 11 Jun 2022cs.AI cs.CV cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite the significant advances achieved in Artificial Neural Networks (ANNs), their design process remains notoriously tedious, depending primarily on intuition, experience and trial-and-error. This human-dependent process is often time-consuming and prone to errors. Furthermore, the models are generally bound to their training contexts, with no considerations to their surrounding environments. Continual adaptiveness and automation of neural networks is of paramount importance to several domains where model accessibility is limited after deployment (e.g IoT devices, self-driving vehicles, etc.). Additionally, even accessible models require frequent maintenance post-deployment to overcome issues such as Concept/Data Drift, which can be cumbersome and restrictive. By leveraging and combining approaches from Neural Architecture Search (NAS) and Continual Learning (CL), more robust and adaptive agents can be developed. This study conducts the first extensive review on the intersection between NAS and CL, formalizing the prospective Continually-Adaptive Neural Networks (CANNs) paradigm and outlining research directions for lifelong autonomous ANNs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 111 of its 247 references that are in the map. First posted on arXiv 11 Jun 2022.

Builds on

111 of 247 references.

References · 111

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2016Learning without Forgettingcited by 2,496
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
38 more
  1. 2016Progressive Neural Networkscited by 1,361
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  5. 2018Experience Replay for Continual Learningcited by 1,095
  6. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  10. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  11. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  12. 2019Gradient based sample selection for online continual learningcited by 571
  13. 2017Progressive Neural Architecture Searchcited by 487
  14. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  15. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  16. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  17. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  18. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  19. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  20. 2017Large-Scale Evolution of Image Classifierscited by 348
  21. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  22. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  23. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  24. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  25. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  26. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
  27. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  28. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  29. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 232
  30. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 202
  31. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 195
  32. 2017Hierarchical Representations for Efficient Architecture Searchcited by 191
  33. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 179
  34. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  35. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  36. 2018Neural Architecture Optimizationcited by 167
  37. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 153
  38. 2018Towards Robust Evaluations of Continual Learningcited by 148

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1; 2026: 2.

4 papers.

  1. 2025SEAL: Searching Expandable Architectures for Incremental Learningcited by 1
  2. 2026A formal definition and meta-model for a machine theory of mindcited by 1
  3. 2022HiveNAS: Neural Architecture Search using Artificial Bee Colony Optimizationcited by 0
  4. 2026HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Searchcited by 0