# Scale-invariant Learning by Physics Inversion

arXiv: https://arxiv.org/abs/2109.15048 · HTML: https://mlsanity.com/paper/2109.15048/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 68 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Learning to Control PDEs with Differentiable Physics](https://mlsanity.com/paper/2001.07457/) (2020)
- [Deep Learning for Real-time Gravitational Wave Detection and Parameter Estimation: Results with Advanced LIGO Data](https://mlsanity.com/paper/1711.03121/) (2017)
- [KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks](https://mlsanity.com/paper/2107.01739/) (2021)
- [Optimization Methods for Inverse Problems](https://mlsanity.com/paper/1712.00154/) (2017)

## Built on by (most cited first, 6 of 6)

- [Solving Inverse Physics Problems with Score Matching](https://mlsanity.com/paper/2301.10250/) (2023)
- [Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology](https://mlsanity.com/paper/2310.05227/) (2023)
- [Approximating Numerical Fluxes Using Fourier Neural Operators for Hyperbolic Conservation Laws](https://mlsanity.com/paper/2401.01783/) (2024)
- [Metamizer: a versatile neural optimizer for fast and accurate physics simulations](https://mlsanity.com/paper/2410.19746/) (2024)
- [Neural Operators Learn the Local Physics of Magnetohydrodynamics](https://mlsanity.com/paper/2404.16015/) (2024)
- [A spectral clustering-type algorithm for the consistent estimation of the Hurst distribution in moderately high dimensions](https://mlsanity.com/paper/2501.18115/) (2025)

## Abstract (arXiv)

Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science. Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems. We find that state-of-the-art training techniques are not well-suited to many problems that involve physical processes. The highly nonlinear behavior, common in physical processes, results in strongly varying gradients that lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. We propose a novel hybrid training approach that combines higher-order optimization methods with machine learning techniques. We take updates from a scale-invariant inverse problem solver and embed them into the gradient-descent-based learning pipeline, replacing the regular gradient of the physical process. We demonstrate the capabilities of our method on a variety of canonical physical systems, showing that it yields significant improvements on a wide range of optimization and learning problems.