mlsanityData to 14 Oct 2026

Learning where to learn: Training data distribution optimization for scientific machine learning

arXiv 2505.21626 ↗First posted 27 May 2025cs.LG math.OC stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

In scientific machine learning, models are routinely deployed with parameter values or boundary conditions far from those used in training. This paper studies the learning-where-to-learn problem of designing a training data distribution that minimizes average prediction error across a family of deployment regimes. A theoretical analysis shows how the training distribution shapes deployment accuracy. This motivates two adaptive algorithms based on bilevel or alternating optimization in the space of probability measures. Discretized implementations using parametric distribution classes or nonparametric particle-based gradient flows deliver optimized training distributions that outperform nonadaptive designs. Once trained, the resulting models exhibit improved sample complexity and robustness to distribution shift. This framework unlocks the potential of principled data acquisition for learning functions and solution operators of partial differential equations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 56 of its 101 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

56 of 101 references.

References · 56

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  3. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  4. 2019Invariant Risk Minimizationcited by 1,854
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  7. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  8. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  9. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 236
  10. 2018Sorting out Lipschitz function approximationcited by 213
  11. 2019Orthogonal Convolutional Neural Networkscited by 123
  12. 2019Variational Bayesian Optimal Experimental Designcited by 116
38 more
  1. 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
  2. 2015Less is More: Nystr"om Computational Regularizationcited by 91
  3. 2021Improved architectures and training algorithms for deep operator networkscited by 86
  4. 2024Operator Learning: Algorithms and Analysiscited by 85
  5. 2019Distributionally Robust Optimization: A Reviewcited by 76
  6. 2023A Mathematical Guide to Operator Learningcited by 76
  7. 2021DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equationscited by 66
  8. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  9. 2023Neural Inverse Operators for Solving PDE Inverse Problemscited by 65
  10. 2022Failure-informed adaptive sampling for PINNscited by 64
  11. 2021Convergence Rates for Learning Linear Operators from Noisy Datacited by 55
  12. 2024Optimal experimental design: Formulations and computationscited by 45
  13. 2018On Fast Leverage Score Sampling and Optimal Learningcited by 41
  14. 2022Discovering and forecasting extreme events via active learning in neural operatorscited by 32
  15. 2022Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluationscited by 32
  16. 2023Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equationcited by 32
  17. 2020Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networkscited by 27
  18. 2023Elliptic PDE learning is provably data-efficientcited by 26
  19. 2024Data Complexity Estimates for Operator Learningcited by 26
  20. 2022Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasetscited by 25
  21. 2023Multi-Resolution Active Learning of Fourier Neural Operatorscited by 22
  22. 2024Active Learning for Neural PDE Solverscited by 22
  23. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  24. 2024Operator Learning Using Random Features: A Tool for Scientific Computingcited by 21
  25. 2025DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic predictioncited by 21
  26. 2024Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networkscited by 20
  27. 2019Solving Electrical Impedance Tomography with Deep Learningcited by 18
  28. 2025Operator Learning: A Statistical Perspectivecited by 17
  29. 2024Minimum-Norm Interpolation Under Covariate Shiftcited by 14
  30. 2024Domain Generalisation via Imprecise Learningcited by 14
  31. 2018Bayesian Learning with Wasserstein Barycenterscited by 12
  32. 2022HARFE: Hard-Ridge Random Feature Expansioncited by 12
  33. 2023Kernel quadrature with randomly pivoted Choleskycited by 11
  34. 2021Interpretable Approximation of High-Dimensional Datacited by 10
  35. 2023Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problemscited by 9
  36. 2023Bayesian Posterior Perturbation Analysis with Integral Probability Metricscited by 7
  37. 2024Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Featurescited by 7
  38. 2025Approximation theory for 1-Lipschitz ResNetscited by 7

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Operator learning for the 2D incompressible Navier-Stokes equations: a conformal prediction approach in the data-scarce regimecited by 1
  2. 2026Into the danger zone: stable extrapolation in high-dimensional function and operator learningcited by 0
  3. 2026Stability of Measure-to-Measure Transformers on Sub-Gaussian Datacited by 0