# Never look back - A modified EnKF method and its application to the training of neural networks without back propagation

arXiv: https://arxiv.org/abs/1805.08034 · HTML: https://mlsanity.com/paper/1805.08034/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 9 of 9)

- [Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversion](https://mlsanity.com/paper/2307.07882/) (2023)
- [Second Order Ensemble Langevin Method for Sampling and Inverse Problems](https://mlsanity.com/paper/2208.04506/) (2022)
- [Efficient Bayesian Physics Informed Neural Networks for Inverse Problems via Ensemble Kalman Inversion](https://mlsanity.com/paper/2303.07392/) (2023)
- [Stable Anderson Acceleration for Deep Learning](https://mlsanity.com/paper/2110.14813/) (2021)
- [Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI](https://mlsanity.com/paper/2411.15265/) (2024)
- [Affine Invariant Ensemble Transform Methods to Improve Predictive Uncertainty in Neural Networks](https://mlsanity.com/paper/2309.04742/) (2023)
- [Training neural networks without backpropagation using particles](https://mlsanity.com/paper/2412.05667/) (2024)
- [Ensemble Kalman filter for uncertainty in human language comprehension](https://mlsanity.com/paper/2505.02590/) (2025)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)

## Abstract (arXiv)

In this work, we present a new derivative-free optimization method and investigate its use for training neural networks. Our method is motivated by the Ensemble Kalman Filter (EnKF), which has been used successfully for solving optimization problems that involve large-scale, highly nonlinear dynamical systems. A key benefit of the EnKF method is that it requires only the evaluation of the forward propagation but not its derivatives. Hence, in the context of neural networks, it alleviates the need for back propagation and reduces the memory consumption dramatically. However, the method is not a pure "black-box" global optimization heuristic as it efficiently utilizes the structure of typical learning problems. Promising first results of the EnKF for training deep neural networks have been presented recently by Kovachki and Stuart. We propose an important modification of the EnKF that enables us to prove convergence of our method to the minimizer of a strongly convex function. Our method also bears similarity with implicit filtering and we demonstrate its potential for minimizing highly oscillatory functions using a simple example. Further, we provide numerical examples that demonstrate the potential of our method for training deep neural networks.