# Incremental multi-domain learning with network latent tensor factorization

arXiv: https://arxiv.org/abs/1904.06345 · HTML: https://mlsanity.com/paper/1904.06345/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 16 of 16)

- [Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs](https://mlsanity.com/paper/2310.00120/) (2023)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [Multi-Domain Incremental Learning for Semantic Segmentation](https://mlsanity.com/paper/2110.12205/) (2021)
- [On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformers](https://mlsanity.com/paper/2308.09610/) (2023)
- [Tensor Component Analysis for Interpreting the Latent Space of GANs](https://mlsanity.com/paper/2111.11736/) (2021)
- [Model based Multi-agent Reinforcement Learning with Tensor Decompositions](https://mlsanity.com/paper/2110.14524/) (2021)
- [Reinforcement Learning in Factored Action Spaces using Tensor Decompositions](https://mlsanity.com/paper/2110.14538/) (2021)
- [IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offs](https://mlsanity.com/paper/2305.14782/) (2023)
- [Augmented Bilinear Network for Incremental Multi-Stock Time-Series Classification](https://mlsanity.com/paper/2207.11577/) (2022)
- [Disentangling Transfer and Interference in Multi-Domain Learning](https://mlsanity.com/paper/2107.05445/) (2021)
- [Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks](https://mlsanity.com/paper/2107.10963/) (2021)
- [Causal Deep Learning](https://mlsanity.com/paper/2301.00314/) (2023)
- [Tensor Decomposition based Personalized Federated Learning](https://mlsanity.com/paper/2208.12959/) (2022)
- [Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networks](https://mlsanity.com/paper/2310.06124/) (2023)
- [Effective black box adversarial attack with handcrafted kernels](https://mlsanity.com/paper/2303.13887/) (2023)

## Abstract (arXiv)

The prominence of deep learning, large amount of annotated data and increasingly powerful hardware made it possible to reach remarkable performance for supervised classification tasks, in many cases saturating the training sets. However the resulting models are specialized to a single very specific task and domain. Adapting the learned classification to new domains is a hard problem due to at least three reasons: (1) the new domains and the tasks might be drastically different; (2) there might be very limited amount of annotated data on the new domain and (3) full training of a new model for each new task is prohibitive in terms of computation and memory, due to the sheer number of parameters of deep CNNs. In this paper, we present a method to learn new-domains and tasks incrementally, building on prior knowledge from already learned tasks and without catastrophic forgetting. We do so by jointly parametrizing weights across layers using low-rank Tucker structure. The core is task agnostic while a set of task specific factors are learnt on each new domain. We show that leveraging tensor structure enables better performance than simply using matrix operations. Joint tensor modelling also naturally leverages correlations across different layers. Compared with previous methods which have focused on adapting each layer separately, our approach results in more compact representations for each new task/domain. We apply the proposed method to the 10 datasets of the Visual Decathlon Challenge and show that our method offers on average about 7.5x reduction in number of parameters and competitive performance in terms of both classification accuracy and Decathlon score.