# Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradients

arXiv: https://arxiv.org/abs/2209.07075 · HTML: https://mlsanity.com/paper/2209.07075/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Hyperparameter optimization with approximate gradient](https://mlsanity.com/paper/1602.02355/) (2016)
- [Bilevel Optimization: Convergence Analysis and Enhanced Design](https://mlsanity.com/paper/2010.07962/) (2020)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)
- [On the Iteration Complexity of Hypergradient Computation](https://mlsanity.com/paper/2006.16218/) (2020)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters](https://mlsanity.com/paper/1511.06727/) (2015)
- [Optimal control of PDEs using physics-informed neural networks](https://mlsanity.com/paper/2111.09880/) (2021)
- [Stability and Generalization of Bilevel Programming in Hyperparameter Optimization](https://mlsanity.com/paper/2106.04188/) (2021)
- [Fast PDE-constrained optimization via self-supervised operator learning](https://mlsanity.com/paper/2110.13297/) (2021)
- [Greedy Quasi-Newton Methods with Explicit Superlinear Convergence](https://mlsanity.com/paper/2002.00657/) (2020)
- [Learning Composable Energy Surrogates for PDE Order Reduction](https://mlsanity.com/paper/2005.06549/) (2020)
- [Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/2110.10461/) (2021)
- [Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate](https://mlsanity.com/paper/2106.09019/) (2021)

## Built on by (most cited first, 17 of 17)

- [EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers](https://mlsanity.com/paper/2412.20413/) (2024)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos](https://mlsanity.com/paper/2312.06398/) (2023)
- [Efficient PDE-Constrained optimization under high-dimensional uncertainty using derivative-informed neural operators](https://mlsanity.com/paper/2305.20053/) (2023)
- [FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.10949/) (2025)
- [Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization](https://mlsanity.com/paper/2406.14095/) (2024)
- [TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videos](https://mlsanity.com/paper/2508.09811/) (2025)
- [Precise-Physics Driven Text-to-3D Generation](https://mlsanity.com/paper/2403.12438/) (2024)
- [BiLO: Bilevel Local Operator Learning for PDE Inverse Problems](https://mlsanity.com/paper/2404.17789/) (2024)
- [ParticleGS: Learning Neural Gaussian Particle Dynamics from Videos for Prior-free Physical Motion Extrapolation](https://mlsanity.com/paper/2505.20270/) (2025)
- [Accelerated primal-dual methods with enlarged step sizes and operator learning for nonsmooth optimal control problems](https://mlsanity.com/paper/2307.00296/) (2023)
- [Natural Hypergradient Descent: Algorithm Design, Convergence Analysis, and Parallel Implementation](https://mlsanity.com/paper/2602.10905/) (2026)
- [Harnessing AI for Inverse Partial Differential Equation Problems: Past, Present, and Prospects](https://mlsanity.com/paper/2605.16966/) (2026)
- [The ADMM-PINNs Algorithmic Framework for Nonsmooth PDE-Constrained Optimization: A Deep Learning Approach](https://mlsanity.com/paper/2302.08309/) (2023)
- [Conformal mapping based Physics-informed neural networks for designing neutral inclusions](https://mlsanity.com/paper/2501.07809/) (2025)
- [A Single-Loop Bilevel Deep Learning Method for Optimal Control of Obstacle Problems](https://mlsanity.com/paper/2601.04120/) (2026)
- [FlowSymm: Physics Aware, Symmetry Preserving Graph Attention for Network Flow Completion](https://mlsanity.com/paper/2601.22317/) (2026)

## Abstract (arXiv)

Deep learning based approaches like Physics-informed neural networks (PINNs) and DeepONets have shown promise on solving PDE constrained optimization (PDECO) problems. However, existing methods are insufficient to handle those PDE constraints that have a complicated or nonlinear dependency on optimization targets. In this paper, we present a novel bi-level optimization framework to resolve the challenge by decoupling the optimization of the targets and constraints. For the inner loop optimization, we adopt PINNs to solve the PDE constraints only. For the outer loop, we design a novel method by using Broyden's method based on the Implicit Function Theorem (IFT), which is efficient and accurate for approximating hypergradients. We further present theoretical explanations and error analysis of the hypergradients computation. Extensive experiments on multiple large-scale and nonlinear PDE constrained optimization problems demonstrate that our method achieves state-of-the-art results compared with strong baselines.