# Learning Sparse Sharing Architectures for Multiple Tasks

arXiv: https://arxiv.org/abs/1911.05034 · HTML: https://mlsanity.com/paper/1911.05034/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 30 of 31)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization](https://mlsanity.com/paper/2105.12002/) (2021)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Dual-Balancing for Multi-Task Learning](https://mlsanity.com/paper/2308.12029/) (2023)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks](https://mlsanity.com/paper/2311.08273/) (2023)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning](https://mlsanity.com/paper/2210.07565/) (2022)
- [One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER](https://mlsanity.com/paper/2301.10410/) (2023)
- [Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning](https://mlsanity.com/paper/2310.10318/) (2023)
- [AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning](https://mlsanity.com/paper/2106.14126/) (2021)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model](https://mlsanity.com/paper/2309.13018/) (2023)
- [Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models](https://mlsanity.com/paper/2412.13488/) (2024)
- [Learning Twofold Heterogeneous Multi-Task by Sharing Similar Convolution Kernel Pairs](https://mlsanity.com/paper/2101.12431/) (2021)
- [Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Task](https://mlsanity.com/paper/2201.02305/) (2022)
- [Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks](https://mlsanity.com/paper/2211.00106/) (2022)
- [Auxiliary Task-based Deep Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2302.14312/) (2023)
- [Less is More -- Towards parsimonious multi-task models using structured sparsity](https://mlsanity.com/paper/2308.12114/) (2023)
- [AdapMTL: Adaptive Pruning Framework for Multitask Learning Model](https://mlsanity.com/paper/2408.03913/) (2024)
- [Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2511.13133/) (2025)
- [UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception](https://mlsanity.com/paper/2602.01594/) (2026)
- [CPMR: Context-Aware Incremental Sequential Recommendation with Pseudo-Multi-Task Learning](https://mlsanity.com/paper/2309.04802/) (2023)
- [Learning Task-Specific Sampling Strategy for Sparse-View CT Reconstruction](https://mlsanity.com/paper/2409.01544/) (2024)
- [NFLAT: Non-Flat-Lattice Transformer for Chinese Named Entity Recognition](https://mlsanity.com/paper/2205.05832/) (2022)
- [Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks](https://mlsanity.com/paper/2301.11560/) (2023)
- [Multi-task Highly Adaptive Lasso](https://mlsanity.com/paper/2301.12029/) (2023)
- [Unified Lattice Graph Fusion for Chinese Named Entity Recognition](https://mlsanity.com/paper/2312.16917/) (2023)
- [Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning](https://mlsanity.com/paper/2501.12115/) (2025)
- [ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters](https://mlsanity.com/paper/2510.18431/) (2025)

## Abstract (arXiv)

Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanism depends on the relations among the tasks, which is not easy since it is difficult to understand the underlying shared factors among these tasks. In this paper, we propose a novel parameter sharing mechanism, named Sparse Sharing. Given multiple tasks, our approach automatically finds a sparse sharing structure. We start with an over-parameterized base network, from which each task extracts a subnetwork. The subnetworks of multiple tasks are partially overlapped and trained in parallel. We show that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework. We conduct extensive experiments on three sequence labeling tasks. Compared with single-task models and three typical multi-task learning baselines, our proposed approach achieves consistent improvement while requiring fewer parameters.