# PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

arXiv: https://arxiv.org/abs/1711.05769 · HTML: https://mlsanity.com/paper/1711.05769/

## Key facts

- Cited by 898 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 30 of 898)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [Generalized Source-free Domain Adaptation](https://mlsanity.com/paper/2108.01614/) (2021)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Continual Learning and Private Unlearning](https://mlsanity.com/paper/2203.12817/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning](https://mlsanity.com/paper/2402.13669/) (2024)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Continual Learning for Text Classification with Information Disentanglement Based Regularization](https://mlsanity.com/paper/2104.05489/) (2021)
- [Invariant Information Bottleneck for Domain Generalization](https://mlsanity.com/paper/2106.06333/) (2021)
- [On Learning the Geodesic Path for Incremental Learning](https://mlsanity.com/paper/2104.08572/) (2021)
- [Continual Learning of Natural Language Processing Tasks: A Survey](https://mlsanity.com/paper/2211.12701/) (2022)

## Abstract (arXiv)

This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks. By performing iterative pruning and network re-training, we are able to sequentially "pack" multiple tasks into a single network while ensuring minimal drop in performance and minimal storage overhead. Unlike prior work that uses proxy losses to maintain accuracy on older tasks, we always optimize for the task at hand. We perform extensive experiments on a variety of network architectures and large-scale datasets, and observe much better robustness against catastrophic forgetting than prior work. In particular, we are able to add three fine-grained classification tasks to a single ImageNet-trained VGG-16 network and achieve accuracies close to those of separately trained networks for each task. Code available at https://github.com/arunmallya/packnet