# MESSY Estimation: Maximum-Entropy based Stochastic and Symbolic densitY Estimation

arXiv: https://arxiv.org/abs/2306.04120 · HTML: https://mlsanity.com/paper/2306.04120/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 94 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [Learning Equations for Extrapolation and Control](https://mlsanity.com/paper/1806.07259/) (2018)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity](https://mlsanity.com/paper/2006.10782/) (2020)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Symbolic Regression via Neural-Guided Genetic Programming Population Seeding](https://mlsanity.com/paper/2111.00053/) (2021)
- [Bayesian Symbolic Regression](https://mlsanity.com/paper/1910.08892/) (2019)
- [Symbolic Regression is NP-hard](https://mlsanity.com/paper/2207.01018/) (2022)
- [SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence](https://mlsanity.com/paper/2006.02509/) (2020)
- [Where are we now? A large benchmark study of recent symbolic regression methods](https://mlsanity.com/paper/1804.09331/) (2018)
- [Learning multi-scale local conditional probability models of images](https://mlsanity.com/paper/2303.02984/) (2023)
- [On minimax density estimation via measure transport](https://mlsanity.com/paper/2207.10231/) (2022)
- [Stein Variational Gradient Descent as Moment Matching](https://mlsanity.com/paper/1810.11693/) (2018)
- [GSR: A Generalized Symbolic Regression Approach](https://mlsanity.com/paper/2205.15569/) (2022)
- [Wavelet Conditional Renormalization Group](https://mlsanity.com/paper/2207.04941/) (2022)
- [A General Model Validation and Testing Tool](https://mlsanity.com/paper/1908.11251/) (2019)
- [Reliable Neural Networks for Regression Uncertainty Estimation](https://mlsanity.com/paper/2109.08213/) (2021)

## Built on by (most cited first, 6 of 6)

- [Enhanced Human-Robot Collaboration using Constrained Probabilistic Human-Motion Prediction](https://mlsanity.com/paper/2310.03314/) (2023)
- [Data-Driven Discovery of PDEs via the Adjoint Method](https://mlsanity.com/paper/2401.17177/) (2024)
- [ISR: Invertible Symbolic Regression](https://mlsanity.com/paper/2405.06848/) (2024)
- [Symbolic Density Estimation: A Decompositional Approach](https://mlsanity.com/paper/2603.27955/) (2026)
- [Symbolic Density Estimation for Discrete Distributions](https://mlsanity.com/paper/2605.21813/) (2026)
- [Recovery-Directed Symbolic Distillation of Neural Likelihoods](https://mlsanity.com/paper/2609.32409/) (2026)

## Abstract (arXiv)

We introduce MESSY estimation, a Maximum-Entropy based Stochastic and Symbolic densitY estimation method. The proposed approach recovers probability density functions symbolically from samples using moments of a Gradient flow in which the ansatz serves as the driving force. In particular, we construct a gradient-based drift-diffusion process that connects samples of the unknown distribution function to a guess symbolic expression. We then show that when the guess distribution has the maximum entropy form, the parameters of this distribution can be found efficiently by solving a linear system of equations constructed using the moments of the provided samples. Furthermore, we use Symbolic regression to explore the space of smooth functions and find optimal basis functions for the exponent of the maximum entropy functional leading to good conditioning. The cost of the proposed method for each set of selected basis functions is linear with the number of samples and quadratic with the number of basis functions. However, the underlying acceptance/rejection procedure for finding optimal and well-conditioned bases adds to the computational cost. We validate the proposed MESSY estimation method against other benchmark methods for the case of a bi-modal and a discontinuous density, as well as a density at the limit of physical realizability. We find that the addition of a symbolic search for basis functions improves the accuracy of the estimation at a reasonable additional computational cost. Our results suggest that the proposed method outperforms existing density recovery methods in the limit of a small to moderate number of samples by providing a low-bias and tractable symbolic description of the unknown density at a reasonable computational cost.