mlsanityData to 14 Oct 2026

A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning

arXiv 2509.14198 ↗First posted 17 Sep 2025cs.LG cs.NA math.NAcited by 6

Not on the map. Why?

Abstract (arXiv)

Residual-based adaptive strategies are widely used in scientific machine learning but remain largely heuristic. We introduce a unifying variational framework that formalizes these methods by integrating convex transformations of the residual. Different transformations correspond to distinct objective functionals: exponential weights target the minimization of uniform error, while linear weights recover the minimization of quadratic error. Within this perspective, adaptive weighting is equivalent to selecting sampling distributions that optimize the primal objective, thereby linking discretization choices directly to error metrics. This principled approach yields three benefits: (1) it enables systematic design of adaptive schemes across norms, (2) reduces discretization error through variance reduction of the loss estimator, and (3) enhances learning dynamics by improving the gradient signal-to-noise ratio. Extending the framework to operator learning, we demonstrate substantial performance gains across optimizers and architectures. Our results provide a theoretical justification of residual-based adaptivity and establish a foundation for principled discretization and training strategies.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 60 of its 106 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

60 of 106 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  4. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  6. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  7. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  8. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  9. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 284
  10. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 236
  11. 2020Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanismcited by 233
  12. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 197
38 more
  1. 2022FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operatorscited by 188
  2. 2022Respecting causality is all you need for training physics-informed neural networkscited by 156
  3. 2021Exact imposition of boundary conditions with distance functions in physics-informed deep neural networkscited by 137
  4. 2021Efficient training of physics-informed neural networks via importance samplingcited by 110
  5. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 104
  6. 2023PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEscited by 87
  7. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 79
  8. 2022Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Samplingcited by 73
  9. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  10. 2022Learning two-phase microstructure evolution using neural operators and autoencoder architecturescited by 72
  11. 2023Residual-based attention in physics-informed neural networkscited by 72
  12. 2024PirateNets: Physics-informed Deep Learning with Residual Adaptive Networkscited by 72
  13. 2020Physics-informed neural network for ultrasound nondestructive quantification of surface breaking crackscited by 68
  14. 2022Failure-informed adaptive sampling for PINNscited by 64
  15. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 56
  16. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 42
  17. 2021Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusioncited by 38
  18. 2024Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networkscited by 38
  19. 2024On the expressiveness and spectral bias of KANscited by 37
  20. 2024Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modelingcited by 36
  21. 2022Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)cited by 35
  22. 2023On the training and generalization of deep operator networkscited by 31
  23. 2024Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?cited by 31
  24. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 31
  25. 2024RiemannONets: Interpretable Neural Operators for Riemann Problemscited by 28
  26. 2024Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamicscited by 28
  27. 2022Investigating molecular transport in the human brain from MRI with physics-informed neural networkscited by 24
  28. 2022Adaptive Self-supervision Algorithms for Physics-informed Neural Networkscited by 24
  29. 2024Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networkscited by 20
  30. 2018Deep Learning of Turbulent Scalar Mixingcited by 17
  31. 2022Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)cited by 17
  32. 2023An adaptive augmented Lagrangian method for training physics and equality constrained artificial neural networkscited by 16
  33. 2022Random Weight Factorization Improves the Training of Continuous Neural Representationscited by 13
  34. 2024Learning in PINNs: Phase transition, diffusion equilibrium, and generalizationcited by 13
  35. 2024KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamicscited by 13
  36. 2024Predicting Crack Nucleation and Propagation in Brittle Materials Using Deep Operator Networks with Diverse Trunk Architecturescited by 12
  37. 2025FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networkscited by 12
  38. 2024Residual-based Attention Physics-informed Neural Networks for Spatio-Temporal Ageing Assessment of Transformers Operated in Renewable Power Plantscited by 11

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networkscited by 11
  2. 2025ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithmscited by 7
  3. 2025Probabilistic Predictions of Process-Induced Deformation in Carbon/Epoxy Composites Using a Deep Operator Networkcited by 3
  4. 2026Lightweight Geometric Adaptation for Training Physics-Informed Neural Networkscited by 1
  5. 2026Residual Attention Physics-Informed Neural Networks for Robust Multiphysics Simulation of Steady-State Electrothermal Energy Systemscited by 0
  6. 2026Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitionscited by 0