# Multi-Task Zipping via Layer-wise Neuron Sharing

arXiv: https://arxiv.org/abs/1805.09791 · HTML: https://mlsanity.com/paper/1805.09791/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 20 of 20)

- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)
- [PLeaS -- Merging Models with Permutations and Least Squares](https://mlsanity.com/paper/2407.02447/) (2024)
- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models](https://mlsanity.com/paper/2309.06619/) (2023)
- [Robot Fleet Learning via Policy Merging](https://mlsanity.com/paper/2310.01362/) (2023)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Forget the Data and Fine-Tuning! Just Fold the Network to Compress](https://mlsanity.com/paper/2502.10216/) (2025)
- [RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics](https://mlsanity.com/paper/2308.15368/) (2023)
- [MIMONet: Multi-Input Multi-Output On-Device Deep Learning](https://mlsanity.com/paper/2307.11962/) (2023)
- [MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inference](https://mlsanity.com/paper/2209.13883/) (2022)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [Compact representations of convolutional neural networks via weight pruning and quantization](https://mlsanity.com/paper/2108.12704/) (2021)
- [Structured Pruning for Multi-Task Deep Neural Networks](https://mlsanity.com/paper/2304.06840/) (2023)
- [CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devices](https://mlsanity.com/paper/2504.09803/) (2025)
- [Smooth Model Compression without Fine-Tuning](https://mlsanity.com/paper/2505.24469/) (2025)

## Abstract (arXiv)

Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both within-model and cross-model to fit in mobile storage and memory. Previous studies focus on squeezing the redundancy within a single neural network. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with <0.5% increase in the test errors for both tasks. The number of iterations to retrain the combined network is at least 17.8 times lower than that of training a single VGG-16 network. Moreover, experiments show that MTZ is also able to effectively merge multiple residual networks.