# Precision Machine Learning

arXiv: https://arxiv.org/abs/2210.13447 · HTML: https://mlsanity.com/paper/2210.13447/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity](https://mlsanity.com/paper/2006.10782/) (2020)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [Solving Statistical Mechanics Using Variational Autoregressive Networks](https://mlsanity.com/paper/1809.10606/) (2018)
- [A Neural Scaling Law from the Dimension of the Data Manifold](https://mlsanity.com/paper/2004.10802/) (2020)
- [SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python](https://mlsanity.com/paper/1907.10121/) (2019)
- [Gradient Boosting Neural Networks: GrowNet](https://mlsanity.com/paper/2002.07971/) (2020)
- [Expressivity of Deep Neural Networks](https://mlsanity.com/paper/2007.04759/) (2020)
- [Neural Network Approximations for Calabi-Yau Metrics](https://mlsanity.com/paper/2012.15821/) (2020)
- [A comparison of deep networks with ReLU activation function and linear spline-type methods](https://mlsanity.com/paper/1804.02253/) (2018)
- [On the rate of convergence of fully connected very deep neural network regression estimates](https://mlsanity.com/paper/1908.11133/) (2019)

## Built on by (most cited first, 14 of 14)

- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [KAN 2.0: Kolmogorov-Arnold Networks Meet Science](https://mlsanity.com/paper/2408.10205/) (2024)
- [KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics](https://mlsanity.com/paper/2407.04192/) (2024)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [Provably safe systems: the only path to controllable AGI](https://mlsanity.com/paper/2309.01933/) (2023)
- [A Neural Scaling Law from Lottery Ticket Ensembling](https://mlsanity.com/paper/2310.02258/) (2023)
- [Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approach](https://mlsanity.com/paper/2310.18702/) (2023)
- [Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision](https://mlsanity.com/paper/2407.17213/) (2024)
- [Probing optimisation in physics-informed neural networks](https://mlsanity.com/paper/2303.15196/) (2023)
- [BWLer: Barycentric Weight Layer Elucidates a Precision-Conditioning Tradeoff for PINNs](https://mlsanity.com/paper/2506.23024/) (2025)
- [Effective Frontiers: A Unification of Neural Scaling Laws](https://mlsanity.com/paper/2602.02593/) (2026)
- [Characterizing 4-string contact interaction using machine learning](https://mlsanity.com/paper/2211.09129/) (2022)
- [HiPreNets: High-Precision Neural Networks through Progressive Training](https://mlsanity.com/paper/2506.15064/) (2025)

## Abstract (arXiv)

We explore unique considerations involved in fitting ML models to data with very high precision, as is often required for science applications. We empirically compare various function approximation methods and study how they scale with increasing parameters and data. We find that neural networks can often outperform classical approximation methods on high-dimensional examples, by auto-discovering and exploiting modular structures therein. However, neural networks trained with common optimizers are less powerful for low-dimensional cases, which motivates us to study the unique properties of neural network loss landscapes and the corresponding optimization challenges that arise in the high precision regime. To address the optimization issue in low dimensions, we develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.