# Rethinking Hard-Parameter Sharing in Multi-Domain Learning

arXiv: https://arxiv.org/abs/2107.11359 · HTML: https://mlsanity.com/paper/2107.11359/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://mlsanity.com/paper/1911.12423/) (2019)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Learning to Branch for Multi-Task Learning](https://mlsanity.com/paper/2006.01895/) (2020)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)
- [Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images](https://mlsanity.com/paper/1512.02017/) (2015)
- [Universal representations:The missing link between faces, text, planktons, and cat breeds](https://mlsanity.com/paper/1701.07275/) (2017)
- [K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning](https://mlsanity.com/paper/1810.10703/) (2018)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing](https://mlsanity.com/paper/1712.02719/) (2017)
- [Deep Elastic Networks with Model Selection for Multi-Task Learning](https://mlsanity.com/paper/1909.04860/) (2019)
- [Multi-Domain Learning and Identity Mining for Vehicle Re-Identification](https://mlsanity.com/paper/2004.10547/) (2020)
- [What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning](https://mlsanity.com/paper/2007.12415/) (2020)
- [PruneNet: Channel Pruning via Global Importance](https://mlsanity.com/paper/2005.11282/) (2020)
- [Unifying Multi-Domain Multi-Task Learning: Tensor and Neural Network Perspectives](https://mlsanity.com/paper/1611.09345/) (2016)

## Built on by (most cited first, 6 of 6)

- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset](https://mlsanity.com/paper/2407.02751/) (2024)
- [On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network](https://mlsanity.com/paper/2301.12353/) (2023)
- [Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networks](https://mlsanity.com/paper/2310.06124/) (2023)
- [Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multi-Task Learning Perspective](https://mlsanity.com/paper/2412.19547/) (2024)
- [Generalized Gradient Descent is a Hypergraph Functor](https://mlsanity.com/paper/2403.19845/) (2024)

## Abstract (arXiv)

Hard parameter sharing in multi-domain learning (MDL) allows domains to share some of the model parameters to reduce storage cost while improving prediction accuracy. One common sharing practice is to share the bottom layers of a deep neural network among domains while using separate top layers for each domain. In this work, we revisit this common practice via an empirical study on image classification tasks from a diverse set of visual domains and make two surprising observations. (1) Using separate bottom-layer parameters could achieve significantly better performance than the common practice and this phenomenon holds with different experimental settings. (2) A multi-domain model with a small proportion of domain-specific parameters from bottom layers can achieve competitive performance with independent models trained on each domain separately. Our observations suggest that people adopt the new strategy of using separate bottom-layer parameters as a stronger baseline for model design in MDL.