# On Computationally Efficient Learning of Exponential Family Distributions

arXiv: https://arxiv.org/abs/2309.06413 · HTML: https://mlsanity.com/paper/2309.06413/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 79 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [A Kernel Test of Goodness of Fit](https://mlsanity.com/paper/1602.02964/) (2016)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Minimum Stein Discrepancy Estimators](https://mlsanity.com/paper/1906.08283/) (2019)
- [Interaction Screening: Efficient and Sample-Optimal Learning of Ising Models](https://mlsanity.com/paper/1605.07252/) (2016)
- [Density Estimation in Infinite Dimensional Exponential Families](https://mlsanity.com/paper/1312.3516/) (2013)
- [Learning deep kernels for exponential family densities](https://mlsanity.com/paper/1811.08357/) (2018)
- [Exponential Family Estimation via Adversarial Dynamics Embedding](https://mlsanity.com/paper/1904.12083/) (2019)
- [Efficient Statistics, in High Dimensions, from Truncated Samples](https://mlsanity.com/paper/1809.03986/) (2018)
- [On Graphical Models via Univariate Exponential Family Distributions](https://mlsanity.com/paper/1301.4183/) (2013)
- [Efficient Learning of Discrete Graphical Models](https://mlsanity.com/paper/1902.00600/) (2019)
- [Estimation of High-Dimensional Graphical Models Using Regularized Score Matching](https://mlsanity.com/paper/1507.00433/) (2015)
- [Learning Some Popular Gaussian Graphical Models without Condition Number Bounds](https://mlsanity.com/paper/1905.01282/) (2019)
- [Estimating Density Models with Truncation Boundaries using Score Matching](https://mlsanity.com/paper/1910.03834/) (2019)
- [Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families](https://mlsanity.com/paper/1506.02564/) (2015)
- [Outlier-Robust Learning of Ising Models Under Dobrushin's Condition](https://mlsanity.com/paper/2102.02171/) (2021)
- [A Computationally Efficient Method for Learning Exponential Family Distributions](https://mlsanity.com/paper/2110.15397/) (2021)
- [On counterfactual inference with unobserved confounding](https://mlsanity.com/paper/2211.08209/) (2022)
- [Fisher Efficient Inference of Intractable Models](https://mlsanity.com/paper/1805.07454/) (2018)
- [On Learning Continuous Pairwise Markov Random Fields](https://mlsanity.com/paper/2010.15031/) (2020)
- [On Semiparametric Exponential Family Graphical Models](https://mlsanity.com/paper/1412.8697/) (2014)
- [Learning Continuous Exponential Families Beyond Gaussian](https://mlsanity.com/paper/2102.09198/) (2021)
- [Vector-Space Markov Random Fields via Exponential Families](https://mlsanity.com/paper/1505.05117/) (2015)
- [A Likelihood Ratio Framework for High Dimensional Semiparametric Regression](https://mlsanity.com/paper/1412.2295/) (2014)

## Built on by (most cited first, 2 of 2)

- [A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation](https://mlsanity.com/paper/2409.18209/) (2024)
- [Causal Inference with Unobserved Confounding: A Mixture Learning Perspective](https://mlsanity.com/paper/2609.23219/) (2026)

## Abstract (arXiv)

We consider the classical problem of learning, with arbitrary accuracy, the natural parameters of a k-parameter truncated textitminimal exponential family from i.i.d. samples in a computationally and statistically efficient manner. We focus on the setting where the support as well as the natural parameters are appropriately bounded. While the traditional maximum likelihood estimator for this class of exponential family is consistent, asymptotically normal, and asymptotically efficient, evaluating it is computationally hard. In this work, we propose a novel loss function and a computationally efficient estimator that is consistent as well as asymptotically normal under mild conditions. We show that, at the population level, our method can be viewed as the maximum likelihood estimation of a re-parameterized distribution belonging to the same class of exponential family. Further, we show that our estimator can be interpreted as a solution to minimizing a particular Bregman score as well as an instance of minimizing the textitsurrogate likelihood. We also provide finite sample guarantees to achieve an error (in ℓ2-norm) of α in the parameter estimation with sample complexity O(sf poly(k)/α2). Our method achives the order-optimal sample complexity of O(sf log(k)/α2) when tailored for node-wise-sparse Markov random fields. Finally, we demonstrate the performance of our estimator via numerical experiments.