# Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting

arXiv: https://arxiv.org/abs/2406.10327 · HTML: https://mlsanity.com/paper/2406.10327/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning](https://mlsanity.com/paper/1602.03409/) (2016)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimality](https://mlsanity.com/paper/2006.10593/) (2020)
- [Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures](https://mlsanity.com/paper/2001.08370/) (2020)
- [SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention](https://mlsanity.com/paper/2402.10198/) (2024)
- [Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps](https://mlsanity.com/paper/2103.05632/) (2021)
- [Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks](https://mlsanity.com/paper/2006.10581/) (2020)
- [Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers](https://mlsanity.com/paper/2010.11750/) (2020)
- [Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)](https://mlsanity.com/paper/2006.06581/) (2020)
- [Theoretical characterization of uncertainty in high-dimensional linear classification](https://mlsanity.com/paper/2202.03295/) (2022)
- [Large Dimensional Analysis and Improvement of Multi Task Learning](https://mlsanity.com/paper/2009.01591/) (2020)
- [Evolution of transfer learning in natural language processing](https://mlsanity.com/paper/1910.07370/) (2019)
- [Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption](https://mlsanity.com/paper/2310.13434/) (2023)
- [Resolvent convergence under heterogeneous second-moment profiles and quadratic-form control](https://mlsanity.com/paper/2109.02644/) (2021)
- [PCA-based Multi Task Learning: a Random Matrix Approach](https://mlsanity.com/paper/2111.00924/) (2021)
- [Asymptotic Bayes risk of semi-supervised multitask learning on Gaussian mixture](https://mlsanity.com/paper/2303.02048/) (2023)
- [Data Augmentation for Multivariate Time Series Classification: An Experimental Study](https://mlsanity.com/paper/2406.06518/) (2024)
- [A Precise Performance Analysis of Support Vector Regression](https://mlsanity.com/paper/2105.10373/) (2021)

## Built on by (most cited first, 8 of 8)

- [User-friendly Foundation Model Adapters for Multivariate Time Series Classification](https://mlsanity.com/paper/2409.12264/) (2024)
- [A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention](https://mlsanity.com/paper/2507.09394/) (2025)
- [Incorporating priors in learning: a random matrix study under a teacher-student framework](https://mlsanity.com/paper/2509.22124/) (2025)
- [Free Random Projection for In-Context Reinforcement Learning](https://mlsanity.com/paper/2504.06983/) (2025)
- [CATS-Linear: Classification Auxiliary Linear Model for Time Series Forecasting](https://mlsanity.com/paper/2510.08661/) (2025)
- [Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networks](https://mlsanity.com/paper/2511.02401/) (2025)
- [A Random Matrix Theory of Masked Self-Supervised Regression](https://mlsanity.com/paper/2601.23208/) (2026)
- [High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning](https://mlsanity.com/paper/2610.02578/) (2026)

## Abstract (arXiv)

In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.