# A Spectral Approach to Gradient Estimation for Implicit Distributions

arXiv: https://arxiv.org/abs/1806.02925 · HTML: https://mlsanity.com/paper/1806.02925/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2018.

## Built on by (most cited first, 30 of 45)

- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Diffusion Models for Causal Discovery via Topological Ordering](https://mlsanity.com/paper/2210.06201/) (2022)
- [Posterior Meta-Replay for Continual Learning](https://mlsanity.com/paper/2103.01133/) (2021)
- [LiBRe: A Practical Bayesian Approach to Adversarial Detection](https://mlsanity.com/paper/2103.14835/) (2021)
- [A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods](https://mlsanity.com/paper/2305.15027/) (2023)
- [Meta-Learning Reliable Priors in the Function Space](https://mlsanity.com/paper/2106.03195/) (2021)
- [On Stein Variational Neural Network Ensembles](https://mlsanity.com/paper/2106.10760/) (2021)
- [Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning](https://mlsanity.com/paper/2205.06342/) (2022)
- [Semi-Implicit Variational Inference via Score Matching](https://mlsanity.com/paper/2308.10014/) (2023)
- [Sampling with Mirrored Stein Operators](https://mlsanity.com/paper/2106.12506/) (2021)
- [Nonparametric Generative Modeling with Conditional Sliced-Wasserstein Flows](https://mlsanity.com/paper/2305.02164/) (2023)
- [Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration](https://mlsanity.com/paper/2310.17153/) (2023)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Feature Space Particle Inference for Neural Network Ensembles](https://mlsanity.com/paper/2206.00944/) (2022)
- [Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression](https://mlsanity.com/paper/2305.00608/) (2023)
- [Deep Variational Implicit Processes](https://mlsanity.com/paper/2206.06720/) (2022)
- [Bridging the Sim-to-Real Gap with Bayesian Inference](https://mlsanity.com/paper/2403.16644/) (2024)
- [Kernel Semi-Implicit Variational Inference](https://mlsanity.com/paper/2405.18997/) (2024)
- [A Skewness-Based Criterion for Addressing Heteroscedastic Noise in Causal Discovery](https://mlsanity.com/paper/2410.06407/) (2024)
- [MARS: Meta-Learning as Score Matching in the Function Space](https://mlsanity.com/paper/2210.13319/) (2022)
- [On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Design](https://mlsanity.com/paper/2308.09888/) (2023)
- [Low-Budget Simulation-Based Inference with Bayesian Neural Networks](https://mlsanity.com/paper/2408.15136/) (2024)
- [Active Slices for Sliced Stein Discrepancy](https://mlsanity.com/paper/2102.03159/) (2021)
- [AutoTransfer: Subject Transfer Learning with Censored Representations on Biosignals Data](https://mlsanity.com/paper/2112.09796/) (2021)
- [Nonparametric Automatic Differentiation Variational Inference with Spline Approximation](https://mlsanity.com/paper/2403.06302/) (2024)
- [Functional Stochastic Gradient MCMC for Bayesian Neural Networks](https://mlsanity.com/paper/2409.16632/) (2024)
- [Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activations](https://mlsanity.com/paper/2410.15777/) (2024)
- [Causal Ordering for Structure Learning from Time Series](https://mlsanity.com/paper/2510.24639/) (2025)

## Abstract (arXiv)

Recently there have been increasing interests in learning and inference with implicit distributions (i.e., distributions without tractable densities). To this end, we develop a gradient estimator for implicit distributions based on Stein's identity and a spectral decomposition of kernel operators, where the eigenfunctions are approximated by the Nystr"om method. Unlike the previous works that only provide estimates at the sample points, our approach directly estimates the gradient function, thus allows for a simple and principled out-of-sample extension. We provide theoretical results on the error bound of the estimator and discuss the bias-variance tradeoff in practice. The effectiveness of our method is demonstrated by applications to gradient-free Hamiltonian Monte Carlo and variational inference with implicit distributions. Finally, we discuss the intuition behind the estimator by drawing connections between the Nystr"om method and kernel PCA, which indicates that the estimator can automatically adapt to the geometry of the underlying distribution.