# PPO-ACT: Proximal Policy Optimization with Adversarial Curriculum Transfer for Spatial Public Goods Games

arXiv: https://arxiv.org/abs/2505.04302 · HTML: https://mlsanity.com/paper/2505.04302/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 60 references that are in the map.
- First posted on arXiv 7 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Statistical physics of human cooperation](https://mlsanity.com/paper/1705.07161/) (2017)
- [MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Trees](https://mlsanity.com/paper/2209.07225/) (2022)
- [Language-based game theory in the age of artificial intelligence](https://mlsanity.com/paper/2403.08944/) (2024)
- [Probabilistic sharing solves the problem of costly punishment](https://mlsanity.com/paper/1408.1945/) (2014)
- [Competition and cooperation among different punishing strategies in the spatial public goods game](https://mlsanity.com/paper/1508.03325/) (2015)
- [Alliance formation with exclusion in the spatial public goods game](https://mlsanity.com/paper/1705.08668/) (2017)

## Built on by (most cited first, 1 of 1)

- [TUC-PPO: Team Utility-Constrained Proximal Policy Optimization for Spatial Public Goods Games](https://mlsanity.com/paper/2507.02675/) (2025)

## Abstract (arXiv)

This study investigates cooperation evolution mechanisms in the spatial public goods game. A novel deep reinforcement learning framework, Proximal Policy Optimization with Adversarial Curriculum Transfer (PPO-ACT), is proposed to model agent strategy optimization in dynamic environments. Traditional evolutionary game models frequently exhibit limitations in modeling long-term decision-making processes. Deep reinforcement learning effectively addresses this limitation by bridging policy gradient methods with evolutionary game theory. Our study pioneers the application of proximal policy optimization's continuous strategy optimization capability to public goods games through a two-stage adversarial curriculum transfer training paradigm. The experimental results show that PPO-ACT performs better in critical enhancement factor regimes. Compared to conventional standard proximal policy optimization methods, Q-learning and Fermi update rules, achieve earlier cooperation phase transitions and maintain stable cooperative equilibria. This framework exhibits better robustness when handling challenging scenarios like all-defector initial conditions. Systematic comparisons reveal the unique advantage of policy gradient methods in population-scale cooperation, i.e., achieving spatiotemporal payoff coordination through value function propagation. Our work provides a new computational framework for studying cooperation emergence in complex systems, algorithmically validating the punishment promotes cooperation hypothesis while offering methodological insights for multi-agent system strategy design.