# Noise Covariance Estimation in Multi-Task High-dimensional Linear Models

arXiv: https://arxiv.org/abs/2206.07256 · HTML: https://mlsanity.com/paper/2206.07256/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 9 of 9)

- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Out-of-sample error estimate for robust M-estimators with convex penalty](https://mlsanity.com/paper/2008.11840/) (2020)
- [Estimating the error variance in a high-dimensional linear model](https://mlsanity.com/paper/1712.02412/) (2017)
- [A Blockwise Descent Algorithm for Group-penalized Multiresponse and Multinomial Regression](https://mlsanity.com/paper/1311.6529/) (2013)
- [CAD: Debiasing the Lasso with inaccurate covariate model](https://mlsanity.com/paper/2107.14172/) (2021)
- [EigenPrism: Inference for High-Dimensional Signal-to-Noise Ratios](https://mlsanity.com/paper/1505.02097/) (2015)
- [First order expansion of convex regularized estimators](https://mlsanity.com/paper/1910.05480/) (2019)
- [New insights for the multivariate square-root lasso](https://mlsanity.com/paper/1909.05041/) (2019)
- [Chi-square and normal inference in high-dimensional multi-task regression](https://mlsanity.com/paper/2107.07828/) (2021)

## Built on by (most cited first, 3 of 3)

- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions](https://mlsanity.com/paper/2305.17825/) (2023)
- [Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression](https://mlsanity.com/paper/2410.02629/) (2024)

## Abstract (arXiv)

This paper studies the multi-task high-dimensional linear regression models where the noise among different tasks is correlated, in the moderately high dimensional regime where sample size n and dimension p are of the same order. Our goal is to estimate the covariance matrix of the noise random vectors, or equivalently the correlation of the noise variables on any pair of two tasks. Treating the regression coefficients as a nuisance parameter, we leverage the multi-task elastic-net and multi-task lasso estimators to estimate the nuisance. By precisely understanding the bias of the squared residual matrix and by correcting this bias, we develop a novel estimator of the noise covariance that converges in Frobenius norm at the rate n-1/2 when the covariates are Gaussian. This novel estimator is efficiently computable. Under suitable conditions, the proposed estimator of the noise covariance attains the same rate of convergence as the "oracle" estimator that knows in advance the regression coefficients of the multi-task model. The Frobenius error bounds obtained in this paper also illustrate the advantage of this new estimator compared to a method-of-moments estimator that does not attempt to estimate the nuisance. As a byproduct of our techniques, we obtain an estimate of the generalization error of the multi-task elastic-net and multi-task lasso estimators. Extensive simulation studies are carried out to illustrate the numerical performance of the proposed method.