The ℓ-test: leveraging sparsity in the Gaussian linear model for improved inference

We develop novel LASSO-based methods for coefficient testing and confidence interval construction in the Gaussian linear model with n≥ d. Our methods' finite-sample validity is identical to that of their ubiquitous ordinary-least-squares-t-test-based analogues, yet have substantially higher power when the true coefficient vector is sparse. In particular, under sparsity our coefficient test, which we call the ℓ-test, performs like the one-sided t-test (despite not being given any information about the sign), and ℓ-test-based confidence intervals are correspondingly shorter than the standard t-test-based intervals. The nature of the ℓ-test directly provides a novel exact adjustment conditional on LASSO selection for post-selection inference, allowing for the construction of post-selection p-values and confidence intervals. None of our methods require resampling or Monte Carlo estimation. We perform a variety of simulations and a real data analysis on an HIV drug resistance data set to demonstrate the benefits of the ℓ-test. We additionally show that the ℓ-test can be applied to a large class of asymptotically Gaussian estimators, dramatically expanding its applicability beyond linear models.