A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data

Recent advances have revealed that the rate of convergence of the expected test error in deep supervised learning decays as a function of the intrinsic dimension and not the dimension d of the input space. Existing literature defines this intrinsic dimension as the Minkowski dimension or the manifold dimension of the support of the underlying probability measures, which often results in sub-optimal rates and unrealistic assumptions. In this paper, we consider supervised deep learning when the response given the explanatory variable is distributed according to an exponential family with a β-H"older smooth mean function. We consider an entropic notion of the intrinsic data-dimension and demonstrate that with n independent and identically distributed samples, the test error scales as tildeOleft(n^-frac2β2β + bard2β(λ)right), where bard2β(λ) is the 2β-entropic dimension of λ, the distribution of the explanatory variables. This improves on the best-known rates. Furthermore, under the assumption of an upper-bounded density of the explanatory variables, we characterize the rate of convergence as tildeOleft( d^frac2lfloorβrfloor(β + d)2β + dn^-frac2β2β + dright), establishing that the dependence on d is not exponential but at most polynomial. We also demonstrate that when the explanatory variable has a lower bounded density, this rate in terms of the number of data samples, is nearly optimal for learning the dependence structure for exponential families.