A Black-Box Debiasing Framework for Conditional Sampling

Conditional sampling is a fundamental task in Bayesian statistics and generative modeling. Consider the problem of sampling from the posterior distribution PX|Y=y^* for some observation y^*, where the likelihood PY|X is known, and we are given n i.i.d. samples D=Xii=1n drawn from an unknown prior distribution πX. Suppose that f(hatπXn) is the distribution of a posterior sample generated by an algorithm (e.g. a conditional generative model or the Bayes rule) when hatπXn is the empirical distribution of the training data. Although averaging over the randomness of the training data D, we have mathbbEDleft(hatπXnright)= πX, we do not have mathbbEDleftf(hatπXn)right= f(πX) due to the nonlinearity of f, leading to a bias. In this paper we propose a black-box debiasing scheme that improves the accuracy of such a naive plug-in approach. For any integer k and under boundedness of the likelihood and smoothness of f, we generate samples hatX(1),dots,hatX(k) and weights w1,dots,wk such that sumi=1kwiP_hatX(i) is a k-th order approximation of f(πX), where the generation process treats f as a black-box. Our generation process achieves higher accuracy when averaged over the randomness of the training data, without degrading the variance, which can be interpreted as improving memorization without compromising generalization in generative models.