mlsanityData to 14 Oct 2026

Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

arXiv 2104.06788 ↗First posted 14 Apr 2021cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper we analyze the classification performance of neural network structures without parametric inference. Making use of neural architecture search, we empirically demonstrate that it is possible to find random weight architectures, a deep prior, that enables a linear classification to perform on par with fully trained deep counterparts. Through ablation experiments, we exclude the possibility of winning a weight initialization lottery and confirm that suitable deep priors do not require additional inference. In an extension to continual learning, we investigate the possibility of catastrophic interference free incremental learning. Under the assumption of classes originating from the same data distribution, a deep prior found on only a subset of classes is shown to allow discrimination of further classes through training of a simple linear classifier.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

25 of 38 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2016Wide Residual Networkscited by 3,070
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  10. 2017Continual Learning with Deep Generative Replaycited by 1,158
  11. 2017Deep Image Priorcited by 1,098
  12. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
13 more
  1. 2015Weight Uncertainty in Neural Networkscited by 554
  2. 2017Large-Scale Evolution of Image Classifierscited by 348
  3. 2017Variational Continual Learningcited by 339
  4. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  5. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  6. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  7. 2018Towards Robust Evaluations of Continual Learningcited by 148
  8. 2020A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learningcited by 87
  9. 2015Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?cited by 41
  10. 2021EEC: Learning to Encode and Regenerate Images for Continual Learningcited by 40
  11. 2019Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognitioncited by 26
  12. 2016A Powerful Generative Model Using Random Weights for the Deep Image Representationcited by 14
  13. 2017Randomness in Deconvolutional Networks for Visual Representationcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 1.

4 papers.

  1. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  2. 2023Effects of Architectures on Continual Semantic Segmentationcited by 3
  3. 2022Incremental Learning with Differentiable Architecture and Forgetting Searchcited by 1
  4. 2025Cognitive Neural Architecture Search Reveals Hierarchical Entailmentcited by 0