# Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives

arXiv: https://arxiv.org/abs/2510.11987 · HTML: https://mlsanity.com/paper/2510.11987/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks](https://mlsanity.com/paper/2104.08426/) (2021)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [PFNN: A Penalty-Free Neural Network Method for Solving a Class of Second-Order Boundary-Value Problems on Complex Geometries](https://mlsanity.com/paper/2004.06490/) (2020)
- [VarNet: Variational Neural Networks for the Solution of Partial Differential Equations](https://mlsanity.com/paper/1912.07443/) (2019)
- [Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?](https://mlsanity.com/paper/2405.04230/) (2024)
- [A deep learning energy method for hyperelasticity and viscoelasticity](https://mlsanity.com/paper/2201.08690/) (2022)
- [Solving engineering eigenvalue problems with neural networks using the Rayleigh quotient](https://mlsanity.com/paper/2506.04375/) (2025)

## Built on by (most cited first, 1 of 1)

- [Boundary condition enforcement with PINNs: a comparative study and verification on 3D geometries](https://mlsanity.com/paper/2512.14941/) (2025)

## Abstract (arXiv)

Second-order methods are emerging as promising alternatives to standard first-order optimizers such as gradient descent and ADAM for training neural networks. Though the advantages of including curvature information in computing optimization steps have been celebrated in the scientific machine learning literature, the only second-order methods that have been studied are quasi-Newton, meaning that the Hessian matrix of the objective function is approximated. Though one would expect only to gain from using the true Hessian in place of its approximation, we show that neural network training reliably fails when relying on exact curvature information. The failure modes provide insight both into the geometry of nonlinear discretizations as well as the distribution of stationary points in the loss landscape, leading us to question the conventional wisdom that the loss landscape is replete with local minima.