# Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

arXiv: https://arxiv.org/abs/2104.06788 · HTML: https://mlsanity.com/paper/2104.06788/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning](https://mlsanity.com/paper/2009.01797/) (2020)
- [Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?](https://mlsanity.com/paper/1504.08291/) (2015)
- [EEC: Learning to Encode and Regenerate Images for Continual Learning](https://mlsanity.com/paper/2101.04904/) (2021)
- [Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition](https://mlsanity.com/paper/1905.12019/) (2019)
- [A Powerful Generative Model Using Random Weights for the Deep Image Representation](https://mlsanity.com/paper/1606.04801/) (2016)
- [Randomness in Deconvolutional Networks for Visual Representation](https://mlsanity.com/paper/1704.00330/) (2017)

## Built on by (most cited first, 4 of 4)

- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Effects of Architectures on Continual Semantic Segmentation](https://mlsanity.com/paper/2302.10718/) (2023)
- [Incremental Learning with Differentiable Architecture and Forgetting Search](https://mlsanity.com/paper/2205.09875/) (2022)
- [Cognitive Neural Architecture Search Reveals Hierarchical Entailment](https://mlsanity.com/paper/2502.11141/) (2025)

## Abstract (arXiv)

In this paper we analyze the classification performance of neural network structures without parametric inference. Making use of neural architecture search, we empirically demonstrate that it is possible to find random weight architectures, a deep prior, that enables a linear classification to perform on par with fully trained deep counterparts. Through ablation experiments, we exclude the possibility of winning a weight initialization lottery and confirm that suitable deep priors do not require additional inference. In an extension to continual learning, we investigate the possibility of catastrophic interference free incremental learning. Under the assumption of classes originating from the same data distribution, a deep prior found on only a subset of classes is shown to allow discrimination of further classes through training of a simple linear classifier.