# Staged Depth Training: A Representation Curriculum for PINNs

arXiv: https://arxiv.org/abs/2609.30299 · HTML: https://mlsanity.com/paper/2609.30299/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Efficient physics-informed neural networks using hash encoding](https://mlsanity.com/paper/2302.13397/) (2023)
- [Stacked networks improve physics-informed training: applications to neural networks and deep operator networks](https://mlsanity.com/paper/2311.06483/) (2023)
- [PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting](https://mlsanity.com/paper/2109.14536/) (2021)
- [Efficient Stagewise Pretraining via Progressive Subnetworks](https://mlsanity.com/paper/2402.05913/) (2024)
- [Enhancing Physics-Informed Neural Networks Through Feature Engineering](https://mlsanity.com/paper/2502.07209/) (2025)
- [LatentPINNs: Generative physics-informed neural networks via a latent representation learning](https://mlsanity.com/paper/2305.07671/) (2023)
- [Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equation](https://mlsanity.com/paper/2309.07401/) (2023)
- [Curriculum-Guided Layer Scaling for Language Model Pretraining](https://mlsanity.com/paper/2506.11389/) (2025)
- [LayerLock: Non-collapsing Representation Learning with Progressive Freezing](https://mlsanity.com/paper/2509.10156/) (2025)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)

## Abstract (arXiv)

Representation quality is a central determinant of PINNs' performance, yet standard training leaves representations to emerge implicitly while fitting the final solution. We introduce representation curriculum, an ordered process in which representations are explicitly learned, transferred independently of their predictors, and progressively refined. We realize it with Staged Depth Training (SDT), which trains a shallow prefix under a temporary physics-informed head, discards the head, and freezes the learned prefix while adding depth, without equation-specific encodings or changes to the final architecture. Across the 20 default forward problems in PINNacle with three backbones, SDT improves 40 of 59 equal-budget problem--backbone cells by at least 5% and remains within that band in the rest, with a 32.8% geometric-mean error reduction on a PirateNet-style backbone. Mechanistic ablations suggest that the gain is not explained by optimizer restarts or shallow warm-starting alone. Representation visualizations and hyperparameter-basin analyses provide diagnostic evidence on representation geometry and local sensitivity to shared hyperparameters. On Poisson--Boltzmann 2D, SDT also more than doubles the fitted depth-scaling exponent for both backbones. These results support representation curriculum as a promising training strategy for improving PINNs while preserving the deployed architecture and inference cost.