# TUC-PPO: Team Utility-Constrained Proximal Policy Optimization for Spatial Public Goods Games

arXiv: https://arxiv.org/abs/2507.02675 · HTML: https://mlsanity.com/paper/2507.02675/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 59 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Statistical physics of human cooperation](https://mlsanity.com/paper/1705.07161/) (2017)
- [Evolutionary dynamics of higher-order interactions in social networks](https://mlsanity.com/paper/2001.10313/) (2020)
- [Mathematical foundations of moral preferences](https://mlsanity.com/paper/2101.08193/) (2021)
- [Probabilistic sharing solves the problem of costly punishment](https://mlsanity.com/paper/1408.1945/) (2014)
- [Competition and cooperation among different punishing strategies in the spatial public goods game](https://mlsanity.com/paper/1508.03325/) (2015)
- [Alliance formation with exclusion in the spatial public goods game](https://mlsanity.com/paper/1705.08668/) (2017)
- [Spatial public goods games with queueing and reputation](https://mlsanity.com/paper/2505.09154/) (2025)
- [PPO-ACT: Proximal Policy Optimization with Adversarial Curriculum Transfer for Spatial Public Goods Games](https://mlsanity.com/paper/2505.04302/) (2025)

## Abstract (arXiv)

We introduce Team Utility-Constrained Proximal Policy Optimization (TUC-PPO), a new deep reinforcement learning framework. It extends Proximal Policy Optimization (PPO) by integrating team welfare objectives specifically for spatial public goods games. Unlike conventional approaches where cooperation emerges indirectly from individual rewards, TUC-PPO instead optimizes a bi-level objective integrating policy gradients and team utility constraints. Consequently, all policy updates explicitly incorporate collective payoff thresholds. The framework preserves PPO's policy gradient core while incorporating constrained optimization through adaptive Lagrangian multipliers. Therefore, decentralized agents dynamically balance selfish and cooperative incentives. The comparative analysis demonstrates superior performance of this constrained deep reinforcement learning approach compared to unmodified PPO and evolutionary game theory baselines. It achieves faster convergence to cooperative equilibria and greater stability against invasion by defectors. The framework formally integrates team objectives into policy updates. This work advances multi-agent deep reinforcement learning for social dilemmas while providing new computational tools for evolutionary game theory research.