The out-of-sample prediction error of the square-root-LASSO and related estimators

We study the classical problem of predicting an outcome variable, Y, using a linear combination of a d-dimensional covariate vector, X. We are interested in linear predictors whose coefficients solve: % beginalign* inf_boldsymbolβ ∈ ℝd left( mathbbE_mathbbPn left[ left(Y-Xtopβ right)r right] right)1/r +δ , ρleft(boldsymbolβright), endalign* where δ>0 is a regularization parameter, ρ:ℝd→ ℝ_+ is a convex penalty function, mathbbPn is the empirical distribution of the data, and r≥ 1. We present three sets of new results. First, we provide conditions under which linear predictors based on these estimators % solve a distributionally robust optimization problem: they minimize the worst-case prediction error over distributions that are close to each other in a type of max-sliced Wasserstein metric. Second, we provide a detailed finite-sample and asymptotic analysis of the statistical properties of the balls of distributions over which the worst-case prediction error is analyzed. Third, we use the distributionally robust optimality and our statistical analysis to present i) an oracle recommendation for the choice of regularization parameter, δ, that guarantees good out-of-sample prediction error; and ii) a test-statistic to rank the out-of-sample performance of two different linear estimators. None of our results rely on sparsity assumptions about the true data generating process; thus, they broaden the scope of use of the square-root lasso and related estimators in prediction problems.