# DiSparse: Disentangled Sparsification for Multitask Model Compression

arXiv: https://arxiv.org/abs/2206.04662 · HTML: https://mlsanity.com/paper/2206.04662/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 46)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://mlsanity.com/paper/1911.12423/) (2019)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)

## Built on by (most cited first, 14 of 14)

- [MDP: Multidimensional Vision Model Pruning with Latency Constraint](https://mlsanity.com/paper/2504.02168/) (2025)
- [Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retraining](https://mlsanity.com/paper/2308.06755/) (2023)
- [Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting](https://mlsanity.com/paper/2312.09148/) (2023)
- [Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint](https://mlsanity.com/paper/2406.12079/) (2024)
- [DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2305.17305/) (2023)
- [Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs](https://mlsanity.com/paper/2305.19844/) (2023)
- [Towards Compute-Optimal Transfer Learning](https://mlsanity.com/paper/2304.13164/) (2023)
- [Pruning for Better Domain Generalizability](https://mlsanity.com/paper/2306.13237/) (2023)
- [AdapMTL: Adaptive Pruning Framework for Multitask Learning Model](https://mlsanity.com/paper/2408.03913/) (2024)
- [Continual Learning through Networks Splitting and Merging with Dreaming-Meta-Weighted Model Fusion](https://mlsanity.com/paper/2312.07082/) (2023)
- [Advancing Weight and Channel Sparsification with Enhanced Saliency](https://mlsanity.com/paper/2502.03658/) (2025)
- [Structured Pruning for Multi-Task Deep Neural Networks](https://mlsanity.com/paper/2304.06840/) (2023)
- [Accelerating Inference of Networks in the Frequency Domain](https://mlsanity.com/paper/2410.04342/) (2024)
- [CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devices](https://mlsanity.com/paper/2504.09803/) (2025)

## Abstract (arXiv)

Despite the popularity of Model Compression and Multitask Learning, how to effectively compress a multitask model has been less thoroughly analyzed due to the challenging entanglement of tasks in the parameter space. In this paper, we propose DiSparse, a simple, effective, and first-of-its-kind multitask pruning and sparse training scheme. We consider each task independently by disentangling the importance measurement and take the unanimous decisions among all tasks when performing parameter pruning and selection. Our experimental results demonstrate superior performance on various configurations and settings compared to popular sparse training and pruning methods. Besides the effectiveness in compression, DiSparse also provides a powerful tool to the multitask learning community. Surprisingly, we even observed better performance than some dedicated multitask learning methods in several cases despite the high model sparsity enforced by DiSparse. We analyzed the pruning masks generated with DiSparse and observed strikingly similar sparse network architecture identified by each task even before the training starts. We also observe the existence of a "watershed" layer where the task relatedness sharply drops, implying no benefits in continued parameters sharing. Our code and models will be available at: https://github.com/SHI-Labs/DiSparse-Multitask-Model-Compression.