# AR-DAE: Towards Unbiased Neural Entropy Gradient Estimation

arXiv: https://arxiv.org/abs/2006.05164 · HTML: https://mlsanity.com/paper/2006.05164/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 19 of 19)

- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Score-based diffusion models for accelerated MRI](https://mlsanity.com/paper/2110.05243/) (2021)
- [Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images](https://mlsanity.com/paper/2106.07009/) (2021)
- [Posterior samples of source galaxies in strong gravitational lenses with score-based priors](https://mlsanity.com/paper/2211.03812/) (2022)
- [Noise Distribution Adaptive Self-Supervised Image Denoising using Tweedie Distribution and Score Matching](https://mlsanity.com/paper/2112.03696/) (2021)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [Score-based Self-supervised MRI Denoising](https://mlsanity.com/paper/2505.05631/) (2025)
- [On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Design](https://mlsanity.com/paper/2308.09888/) (2023)
- [Unsupervised Image Denoising with Score Function](https://mlsanity.com/paper/2304.08384/) (2023)
- [Noise2Score3D: Tweedie's Approach for Unsupervised Point Cloud Denoising](https://mlsanity.com/paper/2503.09283/) (2025)
- [Trained Model in Supervised Deep Learning is a Conditional Risk Minimizer](https://mlsanity.com/paper/2202.03674/) (2022)
- [A Unified View of cGANs with and without Classifiers](https://mlsanity.com/paper/2111.01035/) (2021)
- [On normalization-equivariance properties of supervised and unsupervised denoising methods: a survey](https://mlsanity.com/paper/2402.15352/) (2024)
- [Bayesian Experimental Design via Score Matching](https://mlsanity.com/paper/2607.08335/) (2026)
- [Local Curvature Smoothing with Stein's Identity for Efficient Score Matching](https://mlsanity.com/paper/2412.03962/) (2024)
- [Noise2Score3D:Unsupervised Tweedie's Approach for Point Cloud Denoising](https://mlsanity.com/paper/2502.16826/) (2025)
- [Adversarial and Score-Based CT Denoising: CycleGAN vs Noise2Score](https://mlsanity.com/paper/2511.04083/) (2025)
- [UNICORN: Ultrasound Nakagami Imaging via Score Matching and Adaptation for Assessing Hepatic Steatosis](https://mlsanity.com/paper/2603.16942/) (2026)
- [Score-Based Matching with Target Guidance for Cryo-EM Denoising](https://mlsanity.com/paper/2604.17734/) (2026)

## Abstract (arXiv)

Entropy is ubiquitous in machine learning, but it is in general intractable to compute the entropy of the distribution of an arbitrary continuous random variable. In this paper, we propose the amortized residual denoising autoencoder (AR-DAE) to approximate the gradient of the log density function, which can be used to estimate the gradient of entropy. Amortization allows us to significantly reduce the error of the gradient approximator by approaching asymptotic optimality of a regular DAE, in which case the estimation is in theory unbiased. We conduct theoretical and experimental analyses on the approximation error of the proposed method, as well as extensive studies on heuristics to ensure its robustness. Finally, using the proposed gradient approximator to estimate the gradient of entropy, we demonstrate state-of-the-art performance on density estimation with variational autoencoders and continuous control with soft actor-critic.