# Universal Regular Conditional Distributions

arXiv: https://arxiv.org/abs/2105.07743 · HTML: https://mlsanity.com/paper/2105.07743/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 121 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [First-order Methods for Geodesically Convex Optimization](https://mlsanity.com/paper/1602.06053/) (2016)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [Universal Approximation Theorems for Differentiable Geometric Deep Learning](https://mlsanity.com/paper/2101.05390/) (2021)
- [Equivalence of approximation by convolutional neural networks and fully-connected networks](https://mlsanity.com/paper/1809.00973/) (2018)
- [Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimization](https://mlsanity.com/paper/2104.08708/) (2021)
- [Non-Euclidean Universal Approximation](https://mlsanity.com/paper/2006.02341/) (2020)
- [Deep neural networks, generic universal interpolation, and controlled ODEs](https://mlsanity.com/paper/1908.07838/) (2019)
- [Optimal Stopping via Randomized Neural Networks](https://mlsanity.com/paper/2104.13669/) (2021)
- [Risk bounds for reservoir computing](https://mlsanity.com/paper/1910.13886/) (2019)
- [Convergence of Stochastic Proximal Gradient Algorithm](https://mlsanity.com/paper/1403.5074/) (2014)
- [NOMU: Neural Optimization-based Model Uncertainty](https://mlsanity.com/paper/2102.13640/) (2021)
- [NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation](https://mlsanity.com/paper/1809.00082/) (2018)
- [Low-Rank plus Sparse Decomposition of Covariance Matrices using Neural Network Parametrization](https://mlsanity.com/paper/1908.00461/) (2019)
- [Continuous selections of multivalued mappings](https://mlsanity.com/paper/1401.2257/) (2014)

## Built on by (most cited first, 4 of 4)

- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [Approximation Error Upper and Lower Bounds for H"older Class with Transformers](https://mlsanity.com/paper/2605.07463/) (2026)

## Abstract (arXiv)

We introduce a deep learning model that can universally approximate regular conditional distributions (RCDs). The proposed model operates in three phases: first, it linearizes inputs from a given metric space X to ℝd via a feature map, then a deep feedforward neural network processes these linearized features, and then the network's outputs are then transformed to the 1-Wasserstein space P1(ℝD) via a probabilistic extension of the attention mechanism of Bahdanau et al. (2014). Our model, called the textitprobabilistic transformer (PT), can approximate any continuous function from ℝd  to P1(ℝD) uniformly on compact sets, quantitatively. We identify two ways in which the PT avoids the curse of dimensionality when approximating P1(ℝD)-valued functions. The first strategy builds functions in C(ℝd,P1(ℝD)) which can be efficiently approximated by a PT, uniformly on any given compact subset of ℝd. In the second approach, given any function f in C(ℝd,P1(ℝD)), we build compact subsets of ℝd whereon f can be efficiently approximated by a PT.