# Asymptotics of Linear Regression with Linearly Dependent Data

arXiv: https://arxiv.org/abs/2412.03702 · HTML: https://mlsanity.com/paper/2412.03702/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 46 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2403.20200/) (2024)
- [Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods](https://mlsanity.com/paper/2405.13912/) (2024)
- [Benign Overfitting in Time Series Linear Models with Over-Parameterization](https://mlsanity.com/paper/2204.08369/) (2022)
- [ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data](https://mlsanity.com/paper/2406.11666/) (2024)

## Built on by (most cited first, 5 of 5)

- [On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective](https://mlsanity.com/paper/2505.18346/) (2025)
- [Lasso Universality Under Linearly Dependent Covariates in the Sparse Regime](https://mlsanity.com/paper/2608.08390/) (2026)
- [Generalisation and benign over-fitting for linear regression onto random functional covariates](https://mlsanity.com/paper/2508.13895/) (2025)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch](https://mlsanity.com/paper/2607.03660/) (2026)

## Abstract (arXiv)

In this paper we study the asymptotics of linear regression in settings with non-Gaussian covariates where the covariates exhibit a linear dependency structure, departing from the standard assumption of independence. We model the covariates using stochastic processes with spatio-temporal covariance and analyze the performance of ridge regression in the high-dimensional proportional regime, where the number of samples and feature dimensions grow proportionally. A Gaussian universality theorem is proven, demonstrating that the asymptotics are invariant under replacing the non-Gaussian covariates with Gaussian vectors preserving mean and covariance, for which tools from random matrix theory can be used to derive precise characterizations of the estimation error. The estimation error is characterized by a fixed-point equation involving the spectral properties of the spatio-temporal covariance matrices, enabling efficient computation. We then study optimal regularization, overparameterization, and the double descent phenomenon in the context of dependent data. Simulations validate our theoretical predictions, shedding light on how dependencies influence estimation error and the choice of regularization parameters.