# Multi-task Learning by Leveraging the Semantic Information

arXiv: https://arxiv.org/abs/2103.02546 · HTML: https://mlsanity.com/paper/2103.02546/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Domain Generalization Using a Mixture of Multiple Latent Domains](https://mlsanity.com/paper/1911.07661/) (2019)
- [Partial Adversarial Domain Adaptation](https://mlsanity.com/paper/1808.04205/) (2018)
- [Regularized Learning for Domain Adaptation under Label Shifts](https://mlsanity.com/paper/1903.09734/) (2019)
- [Domain-Symmetric Networks for Adversarial Domain Adaptation](https://mlsanity.com/paper/1904.04663/) (2019)
- [Few-Shot Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.02536/) (2017)
- [Theoretical Analysis of Domain Adaptation with Optimal Transport](https://mlsanity.com/paper/1610.04420/) (2016)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Identifying beneficial task relations for multi-task learning in deep neural networks](https://mlsanity.com/paper/1702.08303/) (2017)
- [Label Efficient Learning of Transferable Representations across Domains and Tasks](https://mlsanity.com/paper/1712.00123/) (2017)
- [A Modulation Module for Multi-task Learning with Applications in Image Retrieval](https://mlsanity.com/paper/1807.06708/) (2018)
- [A Principled Approach for Learning Task Similarity in Multitask Learning](https://mlsanity.com/paper/1903.09109/) (2019)
- [Domain Generalization via Optimal Transport with Metric Similarity Learning](https://mlsanity.com/paper/2007.10573/) (2020)
- [Multi-task Learning of Hierarchical Vision-Language Representation](https://mlsanity.com/paper/1812.00500/) (2018)
- [Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities](https://mlsanity.com/paper/1704.03379/) (2017)
- [Multi-Task Learning with Labeled and Unlabeled Tasks](https://mlsanity.com/paper/1602.06518/) (2016)
- [Beyond H-Divergence: Domain Adaptation Theory With Jensen-Shannon Divergence](https://mlsanity.com/paper/2007.15567/) (2020)
- [Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network](https://mlsanity.com/paper/1406.3474/) (2014)

## Built on by (most cited first, 5 of 5)

- [Learning to Transfer for Traffic Forecasting via Multi-task Learning](https://mlsanity.com/paper/2111.15542/) (2021)
- [Personalised Federated Learning On Heterogeneous Feature Spaces](https://mlsanity.com/paper/2301.11447/) (2023)
- [Gap Minimization for Knowledge Sharing and Transfer](https://mlsanity.com/paper/2201.11231/) (2022)
- [Compressed Hierarchical Representations for Multi-Task Learning and Task Clustering](https://mlsanity.com/paper/2205.15882/) (2022)
- [Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefixes](https://mlsanity.com/paper/2305.13499/) (2023)

## Abstract (arXiv)

One crucial objective of multi-task learning is to align distributions across tasks so that the information between them can be transferred and shared. However, existing approaches only focused on matching the marginal feature distribution while ignoring the semantic information, which may hinder the learning performance. To address this issue, we propose to leverage the label information in multi-task learning by exploring the semantic conditional relations among tasks. We first theoretically analyze the generalization bound of multi-task learning based on the notion of Jensen-Shannon divergence, which provides new insights into the value of label information in multi-task learning. Our analysis also leads to a concrete algorithm that jointly matches the semantic distribution and controls label distribution divergence. To confirm the effectiveness of the proposed method, we first compare the algorithm with several baselines on some benchmarks and then test the algorithms under label space shift conditions. Empirical results demonstrate that the proposed method could outperform most baselines and achieve state-of-the-art performance, particularly showing the benefits under the label shift conditions.