# Ensemble-based gradient inference for particle methods in optimization and sampling

arXiv: https://arxiv.org/abs/2209.15420 · HTML: https://mlsanity.com/paper/2209.15420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 15 of 15)

- [A consensus-based global optimization method for high dimensional machine learning problems](https://mlsanity.com/paper/1909.09249/) (2019)
- [A consensus-based model for global optimization and its mean-field limit](https://mlsanity.com/paper/1604.05648/) (2016)
- [Consensus-Based Optimization Methods Converge Globally](https://mlsanity.com/paper/2103.15130/) (2021)
- [Iterative Ensemble Kalman Methods: A Unified Perspective with Some New Variants](https://mlsanity.com/paper/2010.13299/) (2020)
- [Consensus-Based Optimization on the Sphere: Convergence to Global Minimizers and Machine Learning](https://mlsanity.com/paper/2001.11988/) (2020)
- [Trends in Consensus-based optimization](https://mlsanity.com/paper/2104.01383/) (2021)
- [Inexact Derivative-Free Optimization for Bilevel Learning](https://mlsanity.com/paper/2006.12674/) (2020)
- [From particle swarm optimization to consensus based optimization: stochastic modeling and mean-field limit](https://mlsanity.com/paper/2012.05613/) (2020)
- [On the mean-field limit for the consensus-based optimization](https://mlsanity.com/paper/2105.12919/) (2021)
- [Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law](https://mlsanity.com/paper/2111.08136/) (2021)
- [Consensus-Based Optimization on Hypersurfaces: Well-Posedness and Mean-Field Limit](https://mlsanity.com/paper/2001.11994/) (2020)
- [A Derivative-Free Gauss-Newton Method](https://mlsanity.com/paper/1710.11005/) (2017)
- [Consensus-Based Global Optimization with Personal Best](https://mlsanity.com/paper/2005.07084/) (2020)
- [Efficient Calculation of Regular Simplex Gradients](https://mlsanity.com/paper/1710.01427/) (2017)
- [Consensus based optimization via jump-diffusion stochastic differential equations](https://mlsanity.com/paper/2205.04880/) (2022)

## Abstract (arXiv)

We propose an approach based on function evaluations and Bayesian inference to extract higher-order differential information of objective functions from a given ensemble of particles. Pointwise evaluation V(xi)i of some potential V in an ensemble xii contains implicit information about first or higher order derivatives, which can be made explicit with little computational effort (ensemble-based gradient inference -- EGI). We suggest to use this information for the improvement of established ensemble-based numerical methods for optimization and sampling such as Consensus-based optimization and Langevin-based samplers. Numerical studies indicate that the augmented algorithms are often superior to their gradient-free variants, in particular the augmented methods help the ensembles to escape their initial domain, to explore multimodal, non-Gaussian settings and to speed up the collapse at the end of optimization dynamics. The code for the numerical examples in this manuscript can be found in the paper's Github repository (https://github.com/MercuryBench/ensemble-based-gradient.git).