# A Modulation Module for Multi-task Learning with Applications in Image Retrieval

arXiv: https://arxiv.org/abs/1807.06708 · HTML: https://mlsanity.com/paper/1807.06708/

## Key facts

- Cited by 49 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 17 Jul 2018.

## Built on by (most cited first, 30 of 49)

- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Towards Modular LLMs by Building and Reusing a Library of LoRAs](https://mlsanity.com/paper/2405.11157/) (2024)
- [Multi-Task Dense Prediction via Mixture of Low-Rank Experts](https://mlsanity.com/paper/2403.17749/) (2024)
- [The Perfect Blend: Redefining RLHF with Mixture of Judges](https://mlsanity.com/paper/2409.20370/) (2024)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)
- [Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE](https://mlsanity.com/paper/2311.02684/) (2023)
- [Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives](https://mlsanity.com/paper/2308.02066/) (2023)
- [Neural decoding from stereotactic EEG: accounting for electrode variability across subjects](https://mlsanity.com/paper/2411.10458/) (2024)
- [Visible Watermark Removal via Self-calibrated Localization and Background Refinement](https://mlsanity.com/paper/2108.03581/) (2021)
- [Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentation](https://mlsanity.com/paper/2106.11401/) (2021)
- [Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval](https://mlsanity.com/paper/2204.13919/) (2022)
- [Multi-Task Consistency for Active Learning](https://mlsanity.com/paper/2306.12398/) (2023)
- [MAS: Towards Resource-Efficient Federated Multiple-Task Learning](https://mlsanity.com/paper/2307.11285/) (2023)
- [Probing LLMs for Joint Encoding of Linguistic Categories](https://mlsanity.com/paper/2310.18696/) (2023)
- [Permissioned LLMs: Enforcing Access Control in Large Language Models](https://mlsanity.com/paper/2505.22860/) (2025)
- [X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation](https://mlsanity.com/paper/2203.08764/) (2022)
- [Multi-task Learning by Leveraging the Semantic Information](https://mlsanity.com/paper/2103.02546/) (2021)
- [Contextualizing Meta-Learning via Learning to Decompose](https://mlsanity.com/paper/2106.08112/) (2021)
- [Saliency-Regularized Deep Multi-Task Learning](https://mlsanity.com/paper/2207.01117/) (2022)
- [Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps](https://mlsanity.com/paper/2206.02903/) (2022)
- [Tao: Re-Thinking DL-based Microarchitecture Simulation](https://mlsanity.com/paper/2404.10921/) (2024)
- [Image interpretation by iterative bottom-up top-down processing](https://mlsanity.com/paper/2105.05592/) (2021)
- [Medusa: Universal Feature Learning via Attentional Multitasking](https://mlsanity.com/paper/2204.05698/) (2022)
- [Smart Multi-tenant Federated Learning](https://mlsanity.com/paper/2207.04202/) (2022)
- [Curbing Task Interference using Representation Similarity-Guided Multi-Task Feature Sharing](https://mlsanity.com/paper/2208.09427/) (2022)
- [Grouped Adaptive Loss Weighting for Person Search](https://mlsanity.com/paper/2209.11492/) (2022)
- ["It's a Match!" -- A Benchmark of Task Affinity Scores for Joint Learning](https://mlsanity.com/paper/2301.02873/) (2023)
- [Multi-Task Learning Improves Performance In Deep Argument Mining Models](https://mlsanity.com/paper/2307.01401/) (2023)
- [Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement](https://mlsanity.com/paper/2402.03557/) (2024)

## Abstract (arXiv)

Multi-task learning has been widely adopted in many computer vision tasks to improve overall computation efficiency or boost the performance of individual tasks, under the assumption that those tasks are correlated and complementary to each other. However, the relationships between the tasks are complicated in practice, especially when the number of involved tasks scales up. When two tasks are of weak relevance, they may compete or even distract each other during joint training of shared parameters, and as a consequence undermine the learning of all the tasks. This will raise destructive interference which decreases learning efficiency of shared parameters and lead to low quality loss local optimum w.r.t. shared parameters. To address the this problem, we propose a general modulation module, which can be inserted into any convolutional neural network architecture, to encourage the coupling and feature sharing of relevant tasks while disentangling the learning of irrelevant tasks with minor parameters addition. Equipped with this module, gradient directions from different tasks can be enforced to be consistent for those shared parameters, which benefits multi-task joint training. The module is end-to-end learnable without ad-hoc design for specific tasks, and can naturally handle many tasks at the same time. We apply our approach on two retrieval tasks, face retrieval on the CelebA dataset [1] and product retrieval on the UT-Zappos50K dataset [2, 3], and demonstrate its advantage over other multi-task learning methods in both accuracy and storage efficiency.