# An overview of machine learning-enhanced iterative methods for systems of linear and nonlinear equations

arXiv: https://arxiv.org/abs/2610.07211 · HTML: https://mlsanity.com/paper/2610.07211/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 424 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problems](https://mlsanity.com/paper/1906.06925/) (2019)
- [Learning from Linear Algebra: A Graph Neural Network Approach to Preconditioner Design for Conjugate Gradient Solvers](https://mlsanity.com/paper/2405.15557/) (2024)
- [Preconditioning for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.00531/) (2024)
- [Machine-Learned Preconditioners for Linear Solvers in Geophysical Fluid Flows](https://mlsanity.com/paper/2010.02866/) (2020)
- [Accelerating GMRES with Deep Learning in Real-Time](https://mlsanity.com/paper/2103.10975/) (2021)
- [A coarse space acceleration of deep-DDM](https://mlsanity.com/paper/2112.03732/) (2021)
- [Neural Preconditioning Operator for Efficient PDE Solves](https://mlsanity.com/paper/2502.01337/) (2025)
- [Neural-Initialized Newton: Accelerating Nonlinear Finite Elements via Operator Learning](https://mlsanity.com/paper/2511.06802/) (2025)
- [A Neural-Operator Preconditioned Newton Method for Accelerated Nonlinear Solvers](https://mlsanity.com/paper/2511.08811/) (2025)
- [AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decomposition](https://mlsanity.com/paper/2510.08924/) (2025)
- [PEARL: Preconditioner Enhancement through Actor-critic Reinforcement Learning](https://mlsanity.com/paper/2501.10750/) (2025)
- [Evolving Algebraic Multigrid Methods Using Grammar-Guided Genetic Programming](https://mlsanity.com/paper/2412.05852/) (2024)
- [Multi-Preconditioned LBFGS for Training Finite-Basis PINNs](https://mlsanity.com/paper/2601.08709/) (2026)
- [Layer-Parallel Training for Transformers](https://mlsanity.com/paper/2601.09026/) (2026)
- [Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systems](https://mlsanity.com/paper/2412.07127/) (2024)
- [State-dependent preconditioning for the inner-loop in Variational Data Assimilation using Machine Learning](https://mlsanity.com/paper/2501.04369/) (2025)
- [ReFill: Reinforcement Learning for Fill-In Minimization](https://mlsanity.com/paper/2501.16130/) (2025)
- [One-shot acceleration of transient PDE solvers via online-learned preconditioners](https://mlsanity.com/paper/2509.08765/) (2025)
- [A Flow-rate-conserving CNN-based Domain Decomposition Method for Blood Flow Simulations](https://mlsanity.com/paper/2509.15900/) (2025)
- [Introduction to optimization methods for training SciML models](https://mlsanity.com/paper/2601.10222/) (2026)
- [Preconditioned Physics-Informed Neural Operator Training](https://mlsanity.com/paper/2609.36216/) (2026)

## Abstract (arXiv)

Systems of equations arise in a wide range of scientific and engineering applications. The present work focuses on solvers for general systems of equations, including but not limited to those arising from partial differential equations. These systems can be broadly categorized into linear and nonlinear problems. For large linear systems, iterative solvers are generally preferred over direct methods due to the latter's superlinear growth of computational costs. Although convergence theory is well-developed under certain assumptions on the coefficient matrix, many classes of systems still pose open challenges. These difficulties become even more severe for systems of nonlinear equations, where nonlinear solvers typically rely on repeated linearization. For example, Newton's method may even converge quadratically near the solution; it can also converge slowly or diverge when the initial guess is not chosen appropriately. A wide range of solvers with diverse variants and hyperparameter settings exists, and the development of efficient and robust iterative methods remains an active area of research. Recently, machine learning (ML) techniques have been applied to enhance the efficiency of classical iterative methods while preserving their interpretability and reliability. We refer to these ML-enhanced iterative methods as hybrid iterative methods, in the sense that they combine classical iterative methods with ML. This paper provides a comprehensive overview of state-of-the-art approaches to constructing hybrid iterative methods for systems of both linear and nonlinear equations, while also discussing open challenges and outlining potential directions for future research.