# Instance-Level Task Parameters: A Robust Multi-task Weighting Framework

arXiv: https://arxiv.org/abs/2106.06129 · HTML: https://mlsanity.com/paper/2106.06129/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 38)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [BodyNet: Volumetric Inference of 3D Human Body Shapes](https://mlsanity.com/paper/1804.04875/) (2018)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)
- [Self-supervised Learning of Motion Capture](https://mlsanity.com/paper/1712.01337/) (2017)
- [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](https://mlsanity.com/paper/1611.05377/) (2016)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [Branched Multi-Task Networks: Deciding What Layers To Share](https://mlsanity.com/paper/1904.02920/) (2019)
- [Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network](https://mlsanity.com/paper/1505.07922/) (2015)
- [Many Task Learning with Task Routing](https://mlsanity.com/paper/1903.12117/) (2019)
- [MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning](https://mlsanity.com/paper/1904.08492/) (2019)
- [Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels](https://mlsanity.com/paper/1908.09597/) (2019)

## Built on by (most cited first, 4 of 4)

- [Semi-supervised Multi-task Learning for Semantics and Depth](https://mlsanity.com/paper/2110.07197/) (2021)
- [Contrastive Multi-Task Dense Prediction](https://mlsanity.com/paper/2307.07934/) (2023)
- [Sample-Level Weighting for Multi-Task Learning with Auxiliary Tasks](https://mlsanity.com/paper/2306.04519/) (2023)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)

## Abstract (arXiv)

Recent works have shown that deep neural networks benefit from multi-task learning by learning a shared representation across several related tasks. However, performance of such systems depend on relative weighting between various losses involved during training. Prior works on loss weighting schemes assume that instances are equally easy or hard for all tasks. In order to break this assumption, we let the training process dictate the optimal weighting of tasks for every instance in the dataset. More specifically, we equip every instance in the dataset with a set of learnable parameters (instance-level task parameters) where the cardinality is equal to the number of tasks learned by the model. These parameters model the weighting of each task for an instance. They are updated by gradient descent and do not require hand-crafted rules. We conduct extensive experiments on SURREAL and CityScapes datasets, for human shape and pose estimation, depth estimation and semantic segmentation tasks. In these tasks, our approach outperforms recent dynamic loss weighting approaches, e.g. reducing surface estimation errors by 8.97% on SURREAL. When applied to datasets where one or more tasks can have noisy annotations, the proposed method learns to prioritize learning from clean labels for a given task, e.g. reducing surface estimation errors by up to 60%. We also show that we can reliably detect corrupt labels for a given task as a by-product from learned instance-level task parameters.