# Learning Theory for Distribution Regression

arXiv: https://arxiv.org/abs/1411.2066 · HTML: https://mlsanity.com/paper/1411.2066/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2014.

## Built on by (most cited first, 30 of 48)

- [Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction](https://mlsanity.com/paper/2105.04544/) (2021)
- [Signature Methods in Machine Learning](https://mlsanity.com/paper/2206.14674/) (2022)
- [Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem](https://mlsanity.com/paper/2211.08875/) (2022)
- [Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processes](https://mlsanity.com/paper/2109.03582/) (2021)
- [Deconditional Downscaling with Gaussian Processes](https://mlsanity.com/paper/2105.12909/) (2021)
- [Distribution Regression with Sliced Wasserstein Kernels](https://mlsanity.com/paper/2202.03926/) (2022)
- [SymbolicAI: A framework for logic-based approaches combining generative models and solvers](https://mlsanity.com/paper/2402.00854/) (2024)
- [Demystifying amortized causal discovery with transformers](https://mlsanity.com/paper/2405.16924/) (2024)
- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Machine Learning in Orbit Estimation: a Survey](https://mlsanity.com/paper/2207.08993/) (2022)
- [Statistical Optimality of Divide and Conquer Kernel-based Functional Linear Regression](https://mlsanity.com/paper/2211.10968/) (2022)
- [Sliced-Wasserstein Estimation with Spherical Harmonics as Control Variates](https://mlsanity.com/paper/2402.01493/) (2024)
- [Robust Kernel-based Distribution Regression](https://mlsanity.com/paper/2104.10637/) (2021)
- [Characteristic kernels on Hilbert spaces, Banach spaces, and on sets of measures](https://mlsanity.com/paper/2206.07588/) (2022)
- [Gaussian Processes on Distributions based on Regularized Optimal Transport](https://mlsanity.com/paper/2210.06574/) (2022)
- [Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients](https://mlsanity.com/paper/2402.05639/) (2024)
- [One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operators](https://mlsanity.com/paper/2605.06873/) (2026)
- [Domain Generalization by Functional Regression](https://mlsanity.com/paper/2302.04724/) (2023)
- [Statistical learning on measures: an application to persistence diagrams](https://mlsanity.com/paper/2303.08456/) (2023)
- [Learning Theory of Distribution Regression with Neural Networks](https://mlsanity.com/paper/2307.03487/) (2023)
- [Learning to Embed Distributions via Maximum Kernel Entropy](https://mlsanity.com/paper/2408.00549/) (2024)
- [Regularized least squares learning with heavy-tailed noise is minimax optimal](https://mlsanity.com/paper/2505.14214/) (2025)
- [Generalized Identifiability Bounds for Mixture Models with Grouped Samples](https://mlsanity.com/paper/2207.11164/) (2022)
- [Coefficient-based Regularized Distribution Regression](https://mlsanity.com/paper/2208.12427/) (2022)
- [Distributed Gradient Descent for Functional Learning](https://mlsanity.com/paper/2305.07408/) (2023)
- [Learning Counterfactual Distributions via Kernel Nearest Neighbors](https://mlsanity.com/paper/2410.13381/) (2024)
- [Fast and Scalable Score-Based Kernel Calibration Tests](https://mlsanity.com/paper/2510.14711/) (2025)
- [Sequence Modeling with Spectral Mean Flows](https://mlsanity.com/paper/2510.15366/) (2025)
- [DFNN: A Deep Fr'echet Neural Network Framework for Learning Metric-Space-Valued Responses](https://mlsanity.com/paper/2510.17072/) (2025)
- [Measure-to-measure Regression with Transformers](https://mlsanity.com/paper/2605.28075/) (2026)

## Abstract (arXiv)

We focus on the distribution regression problem: regressing to vector-valued outputs from probability measures. Many important machine learning and statistical tasks fit into this framework, including multi-instance learning and point estimation problems without analytical solution (such as hyperparameter or entropy estimation). Despite the large number of available heuristics in the literature, the inherent two-stage sampled nature of the problem makes the theoretical analysis quite challenging, since in practice only samples from sampled distributions are observable, and the estimates have to rely on similarities computed between sets of points. To the best of our knowledge, the only existing technique with consistency guarantees for distribution regression requires kernel density estimation as an intermediate step (which often performs poorly in practice), and the domain of the distributions to be compact Euclidean. In this paper, we study a simple, analytically computable, ridge regression-based alternative to distribution regression, where we embed the distributions to a reproducing kernel Hilbert space, and learn the regressor from the embeddings to the outputs. Our main contribution is to prove that this scheme is consistent in the two-stage sampled setup under mild conditions (on separable topological domains enriched with kernels): we present an exact computational-statistical efficiency trade-off analysis showing that our estimator is able to match the one-stage sampled minimax optimal rate [Caponnetto and De Vito, 2007; Steinwart et al., 2009]. This result answers a 17-year-old open question, establishing the consistency of the classical set kernel [Haussler, 1999; Gaertner et. al, 2002] in regression. We also cover consistency for more recent kernels on distributions, including those due to [Christmann and Steinwart, 2010].