Universal Regular Conditional Distributions

We introduce a deep learning model that can universally approximate regular conditional distributions (RCDs). The proposed model operates in three phases: first, it linearizes inputs from a given metric space X to ℝd via a feature map, then a deep feedforward neural network processes these linearized features, and then the network's outputs are then transformed to the 1-Wasserstein space P1(ℝD) via a probabilistic extension of the attention mechanism of Bahdanau et al. (2014). Our model, called the textitprobabilistic transformer (PT), can approximate any continuous function from ℝd to P1(ℝD) uniformly on compact sets, quantitatively. We identify two ways in which the PT avoids the curse of dimensionality when approximating P1(ℝD)-valued functions. The first strategy builds functions in C(ℝd,P1(ℝD)) which can be efficiently approximated by a PT, uniformly on any given compact subset of ℝd. In the second approach, given any function f in C(ℝd,P1(ℝD)), we build compact subsets of ℝd whereon f can be efficiently approximated by a PT.