# Neural Networks Trained to Solve Differential Equations Learn General Representations

arXiv: https://arxiv.org/abs/1807.00042 · HTML: https://mlsanity.com/paper/1807.00042/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2018.

## Built on by (most cited first, 16 of 16)

- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [Interpolating between BSDEs and PINNs: deep learning for elliptic and parabolic boundary value problems](https://mlsanity.com/paper/2112.03749/) (2021)
- [Physics-Informed Neural Networks for Quantum Eigenvalue Problems](https://mlsanity.com/paper/2203.00451/) (2022)
- [Extracting Reward Functions from Diffusion Models](https://mlsanity.com/paper/2306.01804/) (2023)
- [Elvet -- a neural network-based differential equation and variational problem solver](https://mlsanity.com/paper/2103.14575/) (2021)
- [Feature Engineering with Regularity Structures](https://mlsanity.com/paper/2108.05879/) (2021)
- [Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations](https://mlsanity.com/paper/2106.01078/) (2021)
- [Neural Control of Parametric Solutions for High-dimensional Evolution PDEs](https://mlsanity.com/paper/2302.00045/) (2023)
- [Approximating High-Dimensional Minimal Surfaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2309.02589/) (2023)
- [A Swarm Variant for the Schr"odinger Solver](https://mlsanity.com/paper/2104.04795/) (2021)
- [Revisiting Hidden Representations in Transfer Learning for Medical Imaging](https://mlsanity.com/paper/2302.08272/) (2023)
- [Semi-Implicit Neural Solver for Time-dependent Partial Differential Equations](https://mlsanity.com/paper/2109.01467/) (2021)
- [Qade: Solving Differential Equations on Quantum Annealers](https://mlsanity.com/paper/2204.03657/) (2022)
- [Physics-Informed Neural Networks for Discovering Localised Eigenstates in Disordered Media](https://mlsanity.com/paper/2305.06802/) (2023)
- [Sampling Complexity of Deep Approximation Spaces](https://mlsanity.com/paper/2312.13379/) (2023)
- [Approximation of Solution Operators for High-dimensional PDEs](https://mlsanity.com/paper/2401.10385/) (2024)

## Abstract (arXiv)

We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we visualize the general representations of the first layers, and interpret them as generalized coordinates over the input domain.