The column measure and Gradient-Free Gradient Boosting

Sparse model selection by structural risk minimization leads to a set of a few predictors, ideally a subset of the true predictors. This selection clearly depends on the underlying loss function tilde L. For linear regression with square loss, the particular (functional) Gradient Boosting variant L2-Boosting excels for its computational efficiency even for very large predictor sets, while still providing suitable estimation consistency. For more general loss functions, functional gradients are not always easily accessible or, like in the case of continuous ranking, need not even exist. To close this gap, starting from column selection frequencies obtained from L2-Boosting, we introduce a loss-dependent ''column measure'' nu(tilde L) which mathematically describes variable selection. The fact that certain variables relevant for a particular loss tilde L never get selected by L2-Boosting is reflected by a respective singular part of nu(tilde L) w.r.t. nu(L2). With this concept at hand, it amounts to a suitable change of measure (accounting for singular parts) to make L2-Boosting select variables according to a different loss tilde L. As a consequence, this opens the bridge to applications of simulational techniques such as various resampling techniques, or rejection sampling, to achieve this change of measure in an algorithmic way.