mlsanityData to 14 Oct 2026

Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting

arXiv 2406.10327 ↗First posted 14 Jun 2024stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

29 of 57 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  5. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  6. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  7. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  8. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  9. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  10. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  11. 2016Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learningcited by 209
  12. 2019Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecastingcited by 161
17 more
  1. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  2. 2020Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimalitycited by 90
  3. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  4. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
  5. 2021Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Mapscited by 41
  6. 2020Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networkscited by 30
  7. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 27
  8. 2020Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)cited by 23
  9. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  10. 2020Large Dimensional Analysis and Improvement of Multi Task Learningcited by 9
  11. 2019Evolution of transfer learning in natural language processingcited by 8
  12. 2023Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumptioncited by 6
  13. 2021Resolvent convergence under heterogeneous second-moment profiles and quadratic-form controlcited by 4
  14. 2021PCA-based Multi Task Learning: a Random Matrix Approachcited by 4
  15. 2023Asymptotic Bayes risk of semi-supervised multitask learning on Gaussian mixturecited by 4
  16. 2024Data Augmentation for Multivariate Time Series Classification: An Experimental Studycited by 3
  17. 2021A Precise Performance Analysis of Support Vector Regressioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2024User-friendly Foundation Model Adapters for Multivariate Time Series Classificationcited by 3
  2. 2025A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attentioncited by 1
  3. 2025Incorporating priors in learning: a random matrix study under a teacher-student frameworkcited by 1
  4. 2025Free Random Projection for In-Context Reinforcement Learningcited by 0
  5. 2025CATS-Linear: Classification Auxiliary Linear Model for Time Series Forecastingcited by 0
  6. 2025Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networkscited by 0
  7. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  8. 2026High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learningcited by 0