# Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality

arXiv: https://arxiv.org/abs/2608.01133 · HTML: https://mlsanity.com/paper/2608.01133/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 2 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [RRM: Robust Reward Model Training Mitigates Reward Hacking](https://mlsanity.com/paper/2409.13156/) (2024)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic](https://mlsanity.com/paper/2105.05701/) (2021)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Evaluating the Performance of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2006.16958/) (2020)
- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [Multi-Agent Reinforcement Learning for Connected and Automated Vehicles Control: Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.11084/) (2023)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios](https://mlsanity.com/paper/2210.02300/) (2022)
- [A Multi-Agent Reinforcement Learning Approach For Safe and Efficient Behavior Planning Of Connected Autonomous Vehicles](https://mlsanity.com/paper/2003.04371/) (2020)
- [Successor-Predecessor Intrinsic Exploration](https://mlsanity.com/paper/2305.15277/) (2023)
- [Checkmating One, by Using Many: Combining Mixture of Experts with MCTS to Improve in Chess](https://mlsanity.com/paper/2401.16852/) (2024)
- [Effective Exploration Based on the Structural Information Principles](https://mlsanity.com/paper/2410.06621/) (2024)
- [Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning via Best Response](https://mlsanity.com/paper/2006.09585/) (2020)
- [A Value Based Parallel Update MCTS Method for Multi-Agent Cooperative Decision Making of Connected and Automated Vehicles](https://mlsanity.com/paper/2409.13783/) (2024)
- [SPformer: A Transformer Based DRL Decision Making Method for Connected Automated Vehicles](https://mlsanity.com/paper/2409.15105/) (2024)
- [Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVs](https://mlsanity.com/paper/2507.12110/) (2025)

## Abstract (arXiv)

Evaluating Multi-Agent Reinforcement Learning (MARL) policies in autonomous driving fundamentally relies on extrinsic statistical indicators (e.g., reward curves and success rates), which often mask intrinsic policy degradation and algorithmic blind spots. To break this black-box evaluation, this letter proposes a novel information-theoretic diagnostic framework. By leveraging a fully converged Monte Carlo Tree Search (MCTS) as an asymptotic oracle, we establish a theoretical ground-truth baseline distribution. We formulate a bounded policy optimality score (Mopt) using the forward KL divergence to rigorously penalize fatal collaborative omissions. Crucially, we semantically decouple this metric into lateral and longitudinal dimensions, creating a granular "semantic microscope". Extensive spatial and temporal diagnostics on state-of-the-art MARL architectures and exploration mechanisms demonstrate that our framework conclusively exposes hidden directional biases, identifies temporal average-policy traps, and transforms heuristic hyperparameter tuning into a visually trackable trajectory optimization. This framework establishes a rigorous, model-agnostic standard for benchmarking intrinsic multi-agent policy quality.