# A Tree-Structured Multi-Task Model Recommender

arXiv: https://arxiv.org/abs/2203.05092 · HTML: https://mlsanity.com/paper/2203.05092/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 10 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://mlsanity.com/paper/1911.12423/) (2019)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [Learning to Branch for Multi-Task Learning](https://mlsanity.com/paper/2006.01895/) (2020)
- [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](https://mlsanity.com/paper/1611.05377/) (2016)
- [Branched Multi-Task Networks: Deciding What Layers To Share](https://mlsanity.com/paper/1904.02920/) (2019)
- [The Global Landscape of Neural Networks: An Overview](https://mlsanity.com/paper/2007.01429/) (2020)
- [Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations](https://mlsanity.com/paper/1809.04766/) (2018)
- [Automated Search for Resource-Efficient Branched Multi-Task Networks](https://mlsanity.com/paper/2008.10292/) (2020)
- [MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning](https://mlsanity.com/paper/2003.14058/) (2020)
- [Regularizing Deep Multi-Task Networks using Orthogonal Gradients](https://mlsanity.com/paper/1912.06844/) (2019)
- [Dynamic Task Weighting Methods for Multi-task Networks in Autonomous Driving Systems](https://mlsanity.com/paper/2001.02223/) (2020)
- [Multi-Task Learning for Dense Prediction Tasks: A Survey](https://mlsanity.com/paper/2004.13379/) (2020)
- [Deep Elastic Networks with Model Selection for Multi-Task Learning](https://mlsanity.com/paper/1909.04860/) (2019)
- [FBNetV5: Neural Architecture Search for Multiple Tasks in One Run](https://mlsanity.com/paper/2111.10007/) (2021)

## Built on by (most cited first, 6 of 6)

- [STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Map](https://mlsanity.com/paper/2307.03374/) (2023)
- [Towards Principled Task Grouping for Multi-Task Learning](https://mlsanity.com/paper/2402.15328/) (2024)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [AdapMTL: Adaptive Pruning Framework for Multitask Learning Model](https://mlsanity.com/paper/2408.03913/) (2024)
- [Attacking All Tasks at Once Using Adversarial Examples in Multi-Task Learning](https://mlsanity.com/paper/2305.12066/) (2023)
- [Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networks](https://mlsanity.com/paper/2310.06124/) (2023)

## Abstract (arXiv)

Tree-structured multi-task architectures have been employed to jointly tackle multiple vision tasks in the context of multi-task learning (MTL). The major challenge is to determine where to branch out for each task given a backbone model to optimize for both task accuracy and computation efficiency. To address the challenge, this paper proposes a recommender that, given a set of tasks and a convolutional neural network-based backbone model, automatically suggests tree-structured multi-task architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training. Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods. Our tree-structured multi-task model recommender is open-sourced and available at https://github.com/zhanglijun95/TreeMTL.