# Kernel Exponential Family Estimation via Doubly Dual Embedding

arXiv: https://arxiv.org/abs/1811.02228 · HTML: https://mlsanity.com/paper/1811.02228/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2018.

## Built on by (most cited first, 16 of 16)

- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic](https://mlsanity.com/paper/2405.00987/) (2024)
- [Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration and Planning](https://mlsanity.com/paper/2210.02087/) (2022)
- [EigenVI: score-based variational inference with orthogonal function expansions](https://mlsanity.com/paper/2410.24054/) (2024)
- [Convex-Concave Min-Max Stackelberg Games](https://mlsanity.com/paper/2110.05192/) (2021)
- [Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks](https://mlsanity.com/paper/2107.05134/) (2021)
- [Pseudo-Spherical Contrastive Divergence](https://mlsanity.com/paper/2111.00780/) (2021)
- [Biological Sequence Kernels with Guaranteed Flexibility](https://mlsanity.com/paper/2304.03775/) (2023)
- [Robust No-Regret Learning in Min-Max Stackelberg Games](https://mlsanity.com/paper/2203.14126/) (2022)
- [Reproducing Kernels and New Approaches in Compositional Data Analysis](https://mlsanity.com/paper/2205.01158/) (2022)
- [Sobolev Space Regularised Pre Density Models](https://mlsanity.com/paper/2307.13763/) (2023)
- [Learning Causally Invariant Reward Functions from Diverse Demonstrations](https://mlsanity.com/paper/2409.08012/) (2024)
- [Denoising Score Matching with Random Fourier Features](https://mlsanity.com/paper/2101.05239/) (2021)
- [Robust Estimation for Kernel Exponential Families with Smoothed Total Variation Distances](https://mlsanity.com/paper/2410.20760/) (2024)
- [Scalable Personalised Item Ranking through Parametric Density Estimation](https://mlsanity.com/paper/2105.04769/) (2021)
- [Kernel Deformed Exponential Families for Sparse Continuous Attention](https://mlsanity.com/paper/2111.01222/) (2021)

## Abstract (arXiv)

We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space. Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the development of a flexible sampling strategy that amortizes the cost of Monte-Carlo sampling in the inference stage. The resulting estimator can be easily generalized to kernel conditional exponential families. We establish a connection between kernel exponential family estimation and MMD-GANs, revealing a new perspective for understanding GANs. Compared to the score matching based estimators, the proposed method improves both memory and time efficiency while enjoying stronger statistical properties, such as fully capturing smoothness in its statistical convergence rate while the score matching estimator appears to saturate. Finally, we show that the proposed estimator empirically outperforms state-of-the-art