# Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow

arXiv: https://arxiv.org/abs/2206.03198 · HTML: https://mlsanity.com/paper/2206.03198/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 75 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Deep learning to represent sub-grid processes in climate models](https://mlsanity.com/paper/1806.04731/) (2018)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [Fast Training of Convolutional Networks through FFTs](https://mlsanity.com/paper/1312.5851/) (2013)
- [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Transfer learning based multi-fidelity physics informed deep neural network](https://mlsanity.com/paper/2005.10614/) (2020)
- [Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations](https://mlsanity.com/paper/1709.00037/) (2017)
- [Convolutional-network models to predict wall-bounded turbulence from wall quantities](https://mlsanity.com/paper/2006.12483/) (2020)
- [A posteriori learning for quasi-geostrophic turbulence parametrization](https://mlsanity.com/paper/2204.03911/) (2022)
- [Adaptive wavelet distillation from neural networks through interpretations](https://mlsanity.com/paper/2107.09145/) (2021)
- [Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks](https://mlsanity.com/paper/2006.10581/) (2020)

## Built on by (most cited first, 20 of 20)

- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [In-Context Operator Learning with Data Prompts for Differential Equation Problems](https://mlsanity.com/paper/2304.07993/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws](https://mlsanity.com/paper/2401.07364/) (2024)
- [Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges](https://mlsanity.com/paper/2306.05014/) (2023)
- [Efficiently Computing Sparse Fourier Transforms of q-ary Functions](https://mlsanity.com/paper/2301.06200/) (2023)
- [Fine-Tune Language Models as Multi-Modal Differential Equation Solvers](https://mlsanity.com/paper/2308.05061/) (2023)
- [Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence](https://mlsanity.com/paper/2606.01470/) (2026)
- [Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision](https://mlsanity.com/paper/2407.17213/) (2024)
- [Deep Learning of Solver-Aware Turbulence Closures from Nudged LES Dynamics](https://mlsanity.com/paper/2604.23874/) (2026)
- [Neural network based generation of a 1-dimensional stochastic field with turbulent velocity statistics](https://mlsanity.com/paper/2211.11580/) (2022)
- [A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields](https://mlsanity.com/paper/2307.16580/) (2023)
- [Guided Unconditional and Conditional Generative Models for Super-Resolution and Inference of Quasi-Geostrophic Turbulence](https://mlsanity.com/paper/2507.00719/) (2025)
- [Electron neural closure for turbulent magnetosheath simulations: energy channels](https://mlsanity.com/paper/2510.00282/) (2025)
- [Differentiable Turbulence: Closure as a partial differential equation constrained optimization](https://mlsanity.com/paper/2307.03683/) (2023)
- [Transferring climate change physical knowledge](https://mlsanity.com/paper/2309.14780/) (2023)
- [Fourier analysis of the physics of transfer learning for data-driven subgrid-scale models of ocean turbulence](https://mlsanity.com/paper/2504.15487/) (2025)
- [An Analytical and AI-discovered Stable, Accurate, and Generalizable Subgrid-scale Closure for Geophysical Turbulence](https://mlsanity.com/paper/2509.20365/) (2025)
- [On the importance of learning non-local dynamics for stable data-driven climate modeling: A 1D gravity wave-QBO testbed](https://mlsanity.com/paper/2407.05224/) (2024)

## Abstract (arXiv)

Transfer learning (TL) is becoming a powerful tool in scientific applications of neural networks (NNs), such as weather/climate prediction and turbulence modeling. TL enables out-of-distribution generalization (e.g., extrapolation in parameters) and effective blending of disparate training sets (e.g., simulations and observations). In TL, selected layers of a NN, already trained for a base system, are re-trained using a small dataset from a target system. For effective TL, we need to know 1) what are the best layers to re-train? and 2) what physics are learned during TL? Here, we present novel analyses and a new framework to address (1)-(2) for a broad range of multi-scale, nonlinear systems. Our approach combines spectral analyses of the systems' data with spectral analyses of convolutional NN's activations and kernels, explaining the inner-workings of TL in terms of the system's nonlinear physics. Using subgrid-scale modeling of several setups of 2D turbulence as test cases, we show that the learned kernels are combinations of low-, band-, and high-pass filters, and that TL learns new filters whose nature is consistent with the spectral differences of base and target systems. We also find the shallowest layers are the best to re-train in these cases, which is against the common wisdom guiding TL in machine learning literature. Our framework identifies the best layer(s) to re-train beforehand, based on physics and NN theory. Together, these analyses explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.