Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Not on the map. Why?
Abstract (arXiv)
Machine learning interatomic potentials (MLIPs) have become a hallmark of AI for scientific simulation. While efforts on new architectures and datasets have led to increasingly accurate and general models, the choice of optimizer for training has largely remained unexplored, defaulting to Adam and its variants in the community. Here, we implement and systematically compare a class of recently proposed matrix-structured optimizers, including Muon, SOAP, and the hybrid SOAP-Muon, for training NequIP and Allegro MLIP models. We find that these optimizers can substantially outperform Adam in both convergence speed and final accuracy. SOAP and SOAP-Muon emerge as robust and consistently strong methods, while Muon only provides partial gains relative to Adam. The improvements are particularly pronounced under partial force supervision. Our results indicate that optimizer choice is an overlooked yet impactful design axis for MLIPs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 57 references that are in the map. First posted on arXiv 2 Jul 2026.
Builds on
25 of 57 references.
References · 25
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
- 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
- 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
- 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
- 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
- 2022Learning Local Equivariant Representations for Large-Scale Atomistic Dynamicscited by 256
- 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
- 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
- 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
- 2025UMA: A Family of Universal Models for Atomscited by 105
- 2022Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulationscited by 95
- 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 56
13 more
- 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
- 2024The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domainscited by 40
- 2023Smooth, exact rotational symmetrization for deep learning on point cloudscited by 39
- 2023Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic sizecited by 33
- 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
- 2025An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networkscited by 22
- 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 18
- 2025High-performance training and inference for deep equivariant interatomic potentialscited by 17
- 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 17
- 2026A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attentioncited by 8
- 2025Training a Foundation Model for Materials on a Budgetcited by 6
- 2026Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learningcited by 4
- 2025DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioningcited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.