A deterministic and computable Bernstein-von Mises theorem

Bernstein-von Mises results (BvM) establish that the Laplace approximation is asymptotically correct in the large-data limit. However, these results are inappropriate for computational purposes since they only hold over most, and not all, datasets and involve hard-to-estimate constants. In this article, I present a new BvM theorem which bounds the Kullback-Leibler (KL) divergence between a fixed log-concave density fleft(boldsymbolθright) and its Laplace approximation. The bound goes to 0 as the higher-derivatives of fleft(boldsymbolθright) tend to 0 and fleft(boldsymbolθright) becomes increasingly Gaussian. The classical BvM theorem in the IID large-data asymptote is recovered as a corollary. Critically, this theorem further suggests a number of computable approximations of the KL divergence with the most promising being: [ KLleft(gLAP,fright)≈frac12Var_boldsymbolθ~ gleft(boldsymbolθright)left(logleft[fleft(boldsymbolθright)right]-logleft[gLAPleft(boldsymbolθright)right]right) ] An empirical investigation of these bounds in the logistic classification model reveals that these approximations are great surrogates for the KL divergence. This result, and future results of a similar nature, could provide a path towards rigorously controlling the error due to the Laplace approximation and more modern approximation methods.