mlsanityData to 14 Oct 2026

Rethinking Hard-Parameter Sharing in Multi-Domain Learning

arXiv 2107.11359 ↗First posted 23 Jul 2021cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Hard parameter sharing in multi-domain learning (MDL) allows domains to share some of the model parameters to reduce storage cost while improving prediction accuracy. One common sharing practice is to share the bottom layers of a deep neural network among domains while using separate top layers for each domain. In this work, we revisit this common practice via an empirical study on image classification tasks from a diverse set of visual domains and make two surprising observations. (1) Using separate bottom-layer parameters could achieve significantly better performance than the common practice and this phenomenon holds with different experimental settings. (2) A multi-domain model with a small proportion of domain-specific parameters from bottom layers can achieve competitive performance with independent models trained on each domain separately. Our observations suggest that people adopt the new strategy of using separate bottom-layer parameters as a stronger baseline for model design in MDL.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 23 Jul 2021.

Builds on

28 of 44 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  12. 2017Learning multiple visual domains with residual adapterscited by 557
16 more
  1. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  2. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  3. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 160
  4. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  5. 2020Learning to Branch for Multi-Task Learningcited by 115
  6. 2017Incremental Learning Through Deep Adaptationcited by 108
  7. 2015Visualizing Deep Convolutional Neural Networks Using Natural Pre-Imagescited by 93
  8. 2017Universal representations:The missing link between faces, text, planktons, and cat breedscited by 45
  9. 2018K for the Price of 1: Parameter-efficient Multi-task and Transfer Learningcited by 34
  10. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 32
  11. 2017Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharingcited by 22
  12. 2019Deep Elastic Networks with Model Selection for Multi-Task Learningcited by 20
  13. 2020Multi-Domain Learning and Identity Mining for Vehicle Re-Identificationcited by 19
  14. 2020What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learningcited by 7
  15. 2020PruneNet: Channel Pruning via Global Importancecited by 3
  16. 2016Unifying Multi-Domain Multi-Task Learning: Tensor and Neural Network Perspectivescited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3.

6 papers.

  1. 2022Neural Network Architecture Beyond Width and Depthcited by 17
  2. 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
  3. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
  4. 2023Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networkscited by 1
  5. 2024Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multi-Task Learning Perspectivecited by 1
  6. 2024Generalized Gradient Descent is a Hypergraph Functorcited by 0