# Learning Twofold Heterogeneous Multi-Task by Sharing Similar Convolution Kernel Pairs

arXiv: https://arxiv.org/abs/2101.12431 · HTML: https://mlsanity.com/paper/2101.12431/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 29 Jan 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://mlsanity.com/paper/1911.12423/) (2019)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)
- [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](https://mlsanity.com/paper/1811.06031/) (2018)
- [Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network](https://mlsanity.com/paper/1505.07922/) (2015)
- [Trace Norm Regularised Deep Multi-Task Learning](https://mlsanity.com/paper/1606.04038/) (2016)
- [Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach](https://mlsanity.com/paper/1706.00906/) (2017)
- [Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition](https://mlsanity.com/paper/1912.08077/) (2019)
- [Learning Sparse Sharing Architectures for Multiple Tasks](https://mlsanity.com/paper/1911.05034/) (2019)
- [BlitzNet: A Real-Time Deep Network for Scene Understanding](https://mlsanity.com/paper/1708.02813/) (2017)
- [Embedding Label Structures for Fine-Grained Feature Representation](https://mlsanity.com/paper/1512.02895/) (2015)
- [Filter Pruning by Switching to Neighboring CNNs with Good Attributes](https://mlsanity.com/paper/1904.03961/) (2019)
- [Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network](https://mlsanity.com/paper/1406.3474/) (2014)
- [Multi-Task Driven Feature Models for Thermal Infrared Tracking](https://mlsanity.com/paper/1911.11384/) (2019)
- [Multitask Learning with Single Gradient Step Update for Task Balancing](https://mlsanity.com/paper/2005.09910/) (2020)
- [Emerging Relation Network and Task Embedding for Multi-Task Regression Problems](https://mlsanity.com/paper/2004.14034/) (2020)
- [Multi-task Learning via Adaptation to Similar Tasks for Mortality Prediction of Diverse Rare Diseases](https://mlsanity.com/paper/2004.05318/) (2020)
- [Heterogeneous Multi-task Metric Learning across Multiple Domains](https://mlsanity.com/paper/1904.04081/) (2019)

## Built on by (most cited first, 2 of 2)

- [Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Task](https://mlsanity.com/paper/2201.02305/) (2022)
- [UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception](https://mlsanity.com/paper/2602.01594/) (2026)

## Abstract (arXiv)

Heterogeneous multi-task learning (HMTL) is an important topic in multi-task learning (MTL). Most existing HMTL methods usually solve either scenario where all tasks reside in the same input (feature) space yet unnecessarily the consistent output (label) space or scenario where their input (feature) spaces are heterogeneous while the output (label) space is consistent. However, to the best of our knowledge, there is limited study on twofold heterogeneous MTL (THMTL) scenario where the input and the output spaces are both inconsistent or heterogeneous. In order to handle this complicated scenario, in this paper, we design a simple and effective multi-task adaptive learning (MTAL) network to learn multiple tasks in such THMTL setting. Specifically, we explore and utilize the inherent relationship between tasks for knowledge sharing from similar convolution kernels in individual layers of the MTAL network. Then in order to realize the sharing, we weightedly aggregate any pair of convolutional kernels with their similarity greater than some threshold ρ, consequently, our model effectively performs cross-task learning while suppresses the intra-redundancy of the entire network. Finally, we conduct end-to-end training. Our experimental results demonstrate the effectiveness of our method in comparison with the state-of-the-art counterparts.