mlsanityData to 7 Oct 2026

DiSparse: Disentangled Sparsification for Multitask Model Compression

arXiv 2206.04662 ↗First posted 9 Jun 2022cs.CV cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Despite the popularity of Model Compression and Multitask Learning, how to effectively compress a multitask model has been less thoroughly analyzed due to the challenging entanglement of tasks in the parameter space. In this paper, we propose DiSparse, a simple, effective, and first-of-its-kind multitask pruning and sparse training scheme. We consider each task independently by disentangling the importance measurement and take the unanimous decisions among all tasks when performing parameter pruning and selection. Our experimental results demonstrate superior performance on various configurations and settings compared to popular sparse training and pruning methods. Besides the effectiveness in compression, DiSparse also provides a powerful tool to the multitask learning community. Surprisingly, we even observed better performance than some dedicated multitask learning methods in several cases despite the high model sparsity enforced by DiSparse. We analyzed the pruning masks generated with DiSparse and observed strikingly similar sparse network architecture identified by each task even before the training starts. We also observe the existence of a "watershed" layer where the task relatedness sharply drops, implying no benefits in continued parameters sharing. Our code and models will be available at: https://github.com/SHI-Labs/DiSparse-Multitask-Model-Compression.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

46 of 59 references.

References · 46

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2017Mask R-CNNcited by 7,259
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
34 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  2. 2019Importance Estimation for Neural Network Pruningcited by 531
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  4. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  5. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  6. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  7. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  8. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  9. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  10. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  11. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  12. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  13. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  14. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  15. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  16. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  17. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  18. 2016HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognitioncited by 103
  19. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  20. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  21. 2019Fast Sparse ConvNetscited by 86
  22. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  23. 2016Training Sparse Neural Networkscited by 73
  24. 2019Any-Precision Deep Neural Networkscited by 39
  25. 2019Regularizing Deep Multi-Task Networks using Orthogonal Gradientscited by 36
  26. 2020A Gradient Flow Framework For Analyzing Network Pruningcited by 35
  27. 2014Efficient and Accurate Approximations of Nonlinear Convolutional Networkscited by 30
  28. 2017BlitzNet: A Real-Time Deep Network for Scene Understandingcited by 29
  29. 2018Multi-Task Zipping via Layer-wise Neuron Sharingcited by 20
  30. 2019Deep Elastic Networks with Model Selection for Multi-Task Learningcited by 20
  31. 2016Deep Cross Residual Learning for Multitask Visual Recognitioncited by 16
  32. 2020On the Transferability of Winning Tickets in Non-Natural Image Datasetscited by 10
  33. 2019Pruning-Aware Merging for Efficient Multitask Inferencecited by 6
  34. 2020Bespoke vs. Pret-`a-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Findingcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 3; 2025: 3.

14 papers.

Sort:

  1. 2023Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retrainingcited by 4
  2. 2023Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splittingcited by 4
  3. 2024Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraintcited by 4
  4. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 4
  5. 2023DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learningcited by 3
  6. 2023Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNscited by 3
  7. 2023Towards Compute-Optimal Transfer Learningcited by 2
  8. 2023Pruning for Better Domain Generalizabilitycited by 2
  9. 2024AdapMTL: Adaptive Pruning Framework for Multitask Learning Modelcited by 2
  10. 2023Continual Learning through Networks Splitting and Merging with Dreaming-Meta-Weighted Model Fusioncited by 1
  11. 2025Advancing Weight and Channel Sparsification with Enhanced Saliencycited by 1
  12. 2023Structured Pruning for Multi-Task Deep Neural Networkscited by 0
2 more
  1. 2024Accelerating Inference of Networks in the Frequency Domaincited by 0
  2. 2025CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devicescited by 0