# Value Approximation for Two-Player General-Sum Differential Games with State Constraints

arXiv: https://arxiv.org/abs/2311.16520 · HTML: https://mlsanity.com/paper/2311.16520/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 69 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [DeepReach: A Deep Learning Approach to High-Dimensional Reachability](https://mlsanity.com/paper/2011.02082/) (2020)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning](https://mlsanity.com/paper/2008.13333/) (2020)
- [Deep Learning of Vortex Induced Vibrations](https://mlsanity.com/paper/1808.08952/) (2018)
- [Convergence of the Deep BSDE Method for Coupled FBSDEs](https://mlsanity.com/paper/1811.01165/) (2018)
- [Reach-SDP: Reachability Analysis of Closed-Loop Systems with Neural Network Controllers via Semidefinite Programming](https://mlsanity.com/paper/2004.07876/) (2020)
- [Reachability Analysis of Neural Feedback Loops](https://mlsanity.com/paper/2108.04140/) (2021)
- [Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations](https://mlsanity.com/paper/1907.05317/) (2019)
- [Efficient Iterative Linear-Quadratic Approximations for Nonlinear Multi-Player General-Sum Differential Games](https://mlsanity.com/paper/1909.04694/) (2019)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions](https://mlsanity.com/paper/2012.03390/) (2020)
- [Reachability Analysis of a General Class of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2207.06531/) (2022)
- [OptimizedDP: An Efficient, User-friendly Library For Optimal Control and Dynamic Programming](https://mlsanity.com/paper/2204.05520/) (2022)
- [Generating Formal Safety Assurances for High-Dimensional Reachability](https://mlsanity.com/paper/2209.12336/) (2022)
- [Bayesian Persuasive Driving](https://mlsanity.com/paper/1809.09735/) (2018)
- [When Shall I Be Empathetic? The Utility of Empathetic Parameter Estimation in Multi-Agent Interactions](https://mlsanity.com/paper/2011.02047/) (2020)
- [Approximating Discontinuous Nash Equilibrial Values of Two-Player General-Sum Differential Games](https://mlsanity.com/paper/2207.01773/) (2022)
- [Seeking Nash equilibrium in non-cooperative differential games](https://mlsanity.com/paper/2003.01098/) (2020)
- [Automated Reachability Analysis of Neural Network-Controlled Systems via Adaptive Polytopes](https://mlsanity.com/paper/2212.07553/) (2022)

## Built on by (most cited first, 3 of 3)

- [Pontryagin Neural Operator for Solving Parametric General-Sum Differential Games](https://mlsanity.com/paper/2401.01502/) (2024)
- [Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.09135/) (2025)
- [Learning Nash Equilibrial Hamiltonian for Two-Player Collision-Avoiding Interactions](https://mlsanity.com/paper/2503.07013/) (2025)

## Abstract (arXiv)

Solving Hamilton-Jacobi-Isaacs (HJI) PDEs numerically enables equilibrial feedback control in two-player differential games, yet faces the curse of dimensionality (CoD). While physics-informed neural networks (PINNs) have shown promise in alleviating CoD in solving PDEs, vanilla PINNs fall short in learning discontinuous solutions due to their sampling nature, leading to poor safety performance of the resulting policies when values are discontinuous due to state or temporal logic constraints. In this study, we explore three potential solutions to this challenge: (1) a hybrid learning method that is guided by both supervisory equilibria and the HJI PDE, (2) a value-hardening method where a sequence of HJIs are solved with increasing Lipschitz constant on the constraint violation penalty, and (3) the epigraphical technique that lifts the value to a higher dimensional state space where it becomes continuous. Evaluations through 5D and 9D vehicle and 13D drone simulations reveal that the hybrid method outperforms others in terms of generalization and safety performance by taking advantage of both the supervisory equilibrium values and costates, and the low cost of PINN loss gradients.