Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regression

We examine the necessity of interpolation in overparameterized models, that is, when achieving optimal predictive risk in machine learning problems requires (nearly) interpolating the training data. In particular, we consider simple overparameterized linear regression y = X θ + w with random design X ∈ ℝn × d under the proportional asymptotics d/n → γ ∈ (1, ∞). We precisely characterize how prediction (test) error necessarily scales with training error in this setting. An implication of this characterization is that as the label noise variance σ2 → 0, any estimator that incurs at least mathsfcσ4 training error for some constant mathsfc is necessarily suboptimal and will suffer growth in excess prediction error at least linear in the training error. Thus, optimal performance requires fitting training data to substantially higher accuracy than the inherent noise floor of the problem.