# Continual and Multi-Task Architecture Search

arXiv: https://arxiv.org/abs/1906.05226 · HTML: https://mlsanity.com/paper/1906.05226/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 22 of 22)

- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search](https://mlsanity.com/paper/2105.11871/) (2021)
- [Recent Advances of Multimodal Continual Learning: A Comprehensive Survey](https://mlsanity.com/paper/2410.05352/) (2024)
- [Building a Subspace of Policies for Scalable Continual Learning](https://mlsanity.com/paper/2211.10445/) (2022)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Continual Learning on Graphs: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2402.11565/) (2024)
- [Continual Graph Learning: A Survey](https://mlsanity.com/paper/2301.12230/) (2023)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Learning Versatile Neural Architectures by Propagating Network Codes](https://mlsanity.com/paper/2103.13253/) (2021)
- [Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search](https://mlsanity.com/paper/2204.05941/) (2022)
- [GP-NAS-ensemble: a model for NAS Performance Prediction](https://mlsanity.com/paper/2301.09231/) (2023)
- [Supernet Training for Federated Image Classification under System Heterogeneity](https://mlsanity.com/paper/2206.01366/) (2022)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Pareto-aware Neural Architecture Generation for Diverse Computational Budgets](https://mlsanity.com/paper/2210.07634/) (2022)
- [Prioritize Alignment in Dataset Distillation](https://mlsanity.com/paper/2408.03360/) (2024)
- [Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning](https://mlsanity.com/paper/2507.07485/) (2025)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing](https://mlsanity.com/paper/2111.04497/) (2021)
- [Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansion](https://mlsanity.com/paper/2408.09838/) (2024)
- [Meta-Learning of NAS for Few-shot Learning in Medical Image Applications](https://mlsanity.com/paper/2203.08951/) (2022)
- [Multi-Task Neural Architecture Search Using Architecture Embedding and Transfer Rank](https://mlsanity.com/paper/2504.00772/) (2025)
- [Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent](https://mlsanity.com/paper/2602.19837/) (2026)

## Abstract (arXiv)

Architecture search is the process of automatically learning the neural model or cell structure that best suits the given task. Recently, this approach has shown promising performance improvements (on language modeling and image classification) with reasonable training speed, using a weight sharing strategy called Efficient Neural Architecture Search (ENAS). In our work, we first introduce a novel continual architecture search (CAS) approach, so as to continually evolve the model parameters during the sequential training of several tasks, without losing performance on previously learned tasks (via block-sparsity and orthogonality constraints), thus enabling life-long learning. Next, we explore a multi-task architecture search (MAS) approach over ENAS for finding a unified, single cell structure that performs well across multiple tasks (via joint controller rewards), and hence allows more generalizable transfer of the cell structure knowledge to an unseen new task. We empirically show the effectiveness of our sequential continual learning and parallel multi-task learning based architecture search approaches on diverse sentence-pair classification tasks (GLUE) and multimodal-generation based video captioning tasks. Further, we present several ablations and analyses on the learned cell structures.