# Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2310.09528 · HTML: https://mlsanity.com/paper/2310.09528/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks](https://mlsanity.com/paper/2006.14599/) (2020)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification](https://mlsanity.com/paper/2004.09031/) (2020)
- [Meta-Learning Sparse Implicit Neural Representations](https://mlsanity.com/paper/2110.14678/) (2021)
- [DPM: A Novel Training Method for Physics-Informed Neural Networks in Extrapolation](https://mlsanity.com/paper/2012.02681/) (2020)
- [HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks](https://mlsanity.com/paper/2111.01008/) (2021)
- [Meta-Learning Sparse Compression Networks](https://mlsanity.com/paper/2205.08957/) (2022)
- [SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition](https://mlsanity.com/paper/2211.08760/) (2022)

## Built on by (most cited first, 30 of 44)

- [Parameterized Physics-informed Neural Networks for Parameterized PDEs](https://mlsanity.com/paper/2408.09446/) (2024)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer](https://mlsanity.com/paper/2312.12467/) (2023)
- [Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation](https://mlsanity.com/paper/2502.00782/) (2025)
- [CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations](https://mlsanity.com/paper/2402.14646/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [OmniArch: Building Foundation Model For Scientific Computing](https://mlsanity.com/paper/2402.16014/) (2024)
- [Physics-Regularized Multi-Modal Image Assimilation for Brain Tumor Localization](https://mlsanity.com/paper/2409.20409/) (2024)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions](https://mlsanity.com/paper/2509.05117/) (2025)
- [Towards a Physics Foundation Model](https://mlsanity.com/paper/2509.13805/) (2025)
- [GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning](https://mlsanity.com/paper/2410.23889/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Advancing Generalizability by the Baldwin Effect](https://mlsanity.com/paper/2312.03243/) (2023)
- [Adaptive Physics-informed Neural Networks: A Survey](https://mlsanity.com/paper/2503.18181/) (2025)
- [Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models](https://mlsanity.com/paper/2503.19081/) (2025)
- [A Low Rank Neural Representation of Entropy Solutions](https://mlsanity.com/paper/2406.05694/) (2024)
- [FastLRNR and Sparse Physics Informed Backpropagation](https://mlsanity.com/paper/2410.04001/) (2024)
- [A Two-Phase Deep Learning Framework for Adaptive Time-Stepping in High-Speed Flow Modeling](https://mlsanity.com/paper/2506.07969/) (2025)
- [F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning](https://mlsanity.com/paper/2509.23173/) (2025)
- [Fourier-Modulated Implicit Neural Representation for Multispectral Satellite Image Compression](https://mlsanity.com/paper/2506.01234/) (2025)
- [Neural Functions for Learning Periodic Signal](https://mlsanity.com/paper/2506.09526/) (2025)
- [Meta-learning Structure-Preserving Dynamics](https://mlsanity.com/paper/2508.11205/) (2025)
- [Conditional neural field for spatial dimension reduction of turbulence data: a comparison study](https://mlsanity.com/paper/2510.25135/) (2025)
- [Neural Parameter Regression for Explicit Representations of PDE Solution Operators](https://mlsanity.com/paper/2403.12764/) (2024)
- [Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations](https://mlsanity.com/paper/2503.21166/) (2025)
- [Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation](https://mlsanity.com/paper/2507.16008/) (2025)
- [Nonlinear model reduction for transport-dominated problems](https://mlsanity.com/paper/2602.01397/) (2026)
- [Compositional Meta-Learning for Mitigating Task Heterogeneity in Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.26999/) (2026)
- [Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEs](https://mlsanity.com/paper/2607.23466/) (2026)
- [Scalable physical source-to-field inference with hypernetworks](https://mlsanity.com/paper/2405.05981/) (2024)

## Abstract (arXiv)

In various engineering and applied science applications, repetitive numerical simulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest a path that potentially opens up a possibility for physics-informed neural networks (PINNs), emerging deep-learning-based solvers, to be considered as one such solver. Although PINNs have pioneered a proper integration of deep-learning and scientific computing, they require repetitive time-consuming training of neural networks, which is not suitable for many-query scenarios. To address this issue, we propose a lightweight low-rank PINNs containing only hundreds of model parameters and an associated hypernetwork-based meta-learning algorithm, which allows efficient approximation of solutions of PDEs for varying ranges of PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as "failure modes" of PINNs.