# Learning Compact Neural Networks with Deep Overparameterised Multitask Learning

arXiv: https://arxiv.org/abs/2308.13300 · HTML: https://mlsanity.com/paper/2308.13300/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 16 of 16)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks](https://mlsanity.com/paper/1811.10495/) (2018)
- [DO-Conv: Depthwise Over-parameterized Convolutional Layer](https://mlsanity.com/paper/2006.12030/) (2020)

## Abstract (arXiv)

Compact neural network offers many benefits for real-world applications. However, it is usually challenging to train the compact neural networks with small parameter sizes and low computational costs to achieve the same or better model performance compared to more complex and powerful architecture. This is particularly true for multitask learning, with different tasks competing for resources. We present a simple, efficient and effective multitask learning overparameterisation neural network design by overparameterising the model architecture in training and sharing the overparameterised model parameters more effectively across tasks, for better optimisation and generalisation. Experiments on two challenging multitask datasets (NYUv2 and COCO) demonstrate the effectiveness of the proposed method across various convolutional networks and parameter sizes.