# Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networks

arXiv: https://arxiv.org/abs/2310.06124 · HTML: https://mlsanity.com/paper/2310.06124/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 58 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 44)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Revisiting Batch Normalization For Practical Domain Adaptation](https://mlsanity.com/paper/1603.04779/) (2016)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature](https://mlsanity.com/paper/1505.00855/) (2015)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://mlsanity.com/paper/1911.12423/) (2019)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understanding](https://mlsanity.com/paper/2203.07997/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Parameter-efficient Model Adaptation for Vision Transformers](https://mlsanity.com/paper/2203.16329/) (2022)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference](https://mlsanity.com/paper/2007.12540/) (2020)

## Built on by (most cited first, 1 of 1)

- [MTL-FNO: A Lightweight Multi-Task Fourier Neural Operator for Sparse Field Reconstruction](https://mlsanity.com/paper/2605.26718/) (2026)

## Abstract (arXiv)

Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after another, using a single unified network. The primary challenge and opportunity lie in leveraging shared information across these tasks and domains to enhance the efficiency of the unified network. The efficiency can be in terms of accuracy, storage cost, computation, or sample complexity. In this paper, we introduce a factorized tensor network (FTN) designed to achieve accuracy comparable to that of independent single-task or single-domain networks, while introducing a minimal number of additional parameters. The FTN approach entails incorporating task- or domain-specific low-rank tensor factors into a shared frozen network derived from a source model. This strategy allows for adaptation to numerous target domains and tasks without encountering catastrophic forgetting. Furthermore, FTN requires a significantly smaller number of task-specific parameters compared to existing methods. We performed experiments on widely used multi-domain and multi-task datasets. We show the experiments on convolutional-based architecture with different backbones and on transformer-based architecture. Our findings indicate that FTN attains similar accuracy as single-task or single-domain methods while using only a fraction of additional parameters per task. The code is available at https://doi.org/10.24433/CO.7519211.v2.