mlsanityData to 7 Oct 2026

Staged Depth Training: A Representation Curriculum for PINNs

arXiv 2609.30299 ↗First posted 21 Aug 2026cs.LG physics.comp-phcited by 0

Not on the map. Why?

Abstract (arXiv)

Representation quality is a central determinant of PINNs' performance, yet standard training leaves representations to emerge implicitly while fitting the final solution. We introduce representation curriculum, an ordered process in which representations are explicitly learned, transferred independently of their predictors, and progressively refined. We realize it with Staged Depth Training (SDT), which trains a shallow prefix under a temporary physics-informed head, discards the head, and freezes the learned prefix while adding depth, without equation-specific encodings or changes to the final architecture. Across the 20 default forward problems in PINNacle with three backbones, SDT improves 40 of 59 equal-budget problem--backbone cells by at least 5% and remains within that band in the rest, with a 32.8% geometric-mean error reduction on a PirateNet-style backbone. Mechanistic ablations suggest that the gain is not explained by optimizer restarts or shallow warm-starting alone. Representation visualizations and hyperparameter-basin analyses provide diagnostic evidence on representation geometry and local sensitivity to shared hyperparameters. On Poisson--Boltzmann 2D, SDT also more than doubles the fitted depth-scaling exponent for both backbones. These results support representation curriculum as a promising training strategy for improving PINNs while preserving the deployed architecture and inference cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

19 of 33 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  3. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  4. 2014Deeply-Supervised Netscited by 364
  5. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  6. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  7. 2023PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEscited by 84
  8. 2024PirateNets: Physics-informed Deep Learning with Residual Adaptive Networkscited by 70
  9. 2022Staged Training for Transformer Language Modelscited by 49
  10. 2023Efficient physics-informed neural networks using hash encodingcited by 21
  11. 2023Stacked networks improve physics-informed training: applications to neural networks and deep operator networkscited by 21
  12. 2021PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splittingcited by 20
7 more
  1. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  2. 2025Enhancing Physics-Informed Neural Networks Through Feature Engineeringcited by 8
  3. 2023LatentPINNs: Generative physics-informed neural networks via a latent representation learningcited by 5
  4. 2023Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equationcited by 5
  5. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  6. 2025LayerLock: Non-collapsing Representation Learning with Progressive Freezingcited by 3
  7. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 1

Built on by

Not cited yet.