# Multi-Task Learning with Covariate-Overlap Regularization

arXiv: https://arxiv.org/abs/2505.24281 · HTML: https://mlsanity.com/paper/2505.24281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 13 of 13)

- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](https://mlsanity.com/paper/1811.06031/) (2018)
- [Optimally tackling covariate shift in RKHS-based nonparametric regression](https://mlsanity.com/paper/2205.02986/) (2022)
- [Adaptive and Robust Multi-Task Learning](https://mlsanity.com/paper/2202.05250/) (2022)
- [Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learning](https://mlsanity.com/paper/1801.09851/) (2018)
- [A linear adjustment based approach to posterior drift in transfer learning](https://mlsanity.com/paper/2111.10841/) (2021)
- [Combining Experimental and Historical Data for Policy Evaluation](https://mlsanity.com/paper/2406.00317/) (2024)
- [Representation Retrieval Learning for Heterogeneous Data Integration](https://mlsanity.com/paper/2503.09494/) (2025)

## Abstract (arXiv)

Multi-task learning improves data efficiency by sharing information across related tasks, but indiscriminate sharing can be harmful when their covariate distributions and response relationships differ. We propose COVariate-ovERlap regularized multi-task learning (COVER) to address covariate and posterior heterogeneity. The model combines a common component function with a shared neural representation and low-dimensional task-specific coefficients. Taskwise second-moment matrices summarize covariate heterogeneity and determine the strength of coefficient integration in each representation direction. We derive a covariate-overlap penalty by minimizing the total squared change in two task predictors when their coefficients are replaced by one auxiliary coefficient. An equivalent auxiliary formulation supports end-to-end training without matrix inversion. An exact fixed-representation bias--variance decomposition quantifies how covariate overlap controls variance reduction and how posterior heterogeneity determines shrinkage bias. Global and localized end-to-end oracle inequalities account for jointly learning the neural functions and estimating the overlap matrices from the same observations. We give explicit neural-network rates and sharpen the stochastic prediction term when the regularized oracle risk and overlap-estimation error are small. Simulations across diverse heterogeneity settings show competitive performance against deep-learning and statistical data-integration methods, with the largest gains under joint heterogeneity. In a GTEx central-nervous-system analysis, COVER achieves the lowest response-averaged prediction error among the compared methods and reveals tissue-pair integration patterns.