# Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization

arXiv: https://arxiv.org/abs/2603.02654 · HTML: https://mlsanity.com/paper/2603.02654/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2026.

## Built on by (most cited first, 2 of 2)

- [Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization](https://mlsanity.com/paper/2607.17924/) (2026)
- [MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Games](https://mlsanity.com/paper/2609.06563/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel framework for multi-agent reinforcement learning that enhances sample efficiency and coordination through accurate per-agent advantage estimation. The core of our approach is Generalized Per-Agent Advantage Estimator (GPAE), which employs a per-agent value iteration operator to compute precise per-agent advantages. This operator enables stable off-policy learning by indirectly estimating values via action probabilities, eliminating the need for direct Q-function estimation. To further refine estimation, we introduce a double-truncated importance sampling ratio scheme. This scheme improves credit assignment for off-policy trajectories by balancing sensitivity to the agent's own policy changes with robustness to non-stationarity from other agents. Experiments on benchmarks demonstrate that our approach outperforms existing approaches, excelling in coordination and sample efficiency for complex scenarios.