# Score-Aware Policy-Gradient and Performance Guarantees using Local Lyapunov Stability

arXiv: https://arxiv.org/abs/2312.02804 · HTML: https://mlsanity.com/paper/2312.02804/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2023.

## Built on by (most cited first, 3 of 3)

- [Admission Control of Quasi-Reversible Queueing Systems: Optimization and Reinforcement Learning](https://mlsanity.com/paper/2505.16353/) (2025)
- [Demonstration of effective UCB-based routing in skill-based queues on real-world data](https://mlsanity.com/paper/2506.20543/) (2025)
- [Multi-Robot Multi-Queue Control via Exhaustive Assignment Actor-Critic Learning](https://mlsanity.com/paper/2604.03605/) (2026)

## Abstract (arXiv)

In this paper, we introduce a policy-gradient method for model-based reinforcement learning (RL) that exploits a type of stationary distributions commonly obtained from Markov decision processes (MDPs) in stochastic networks, queueing systems, and statistical mechanics. Specifically, when the stationary distribution of the MDP belongs to an exponential family that is parametrized by policy parameters, we can improve existing policy gradient methods for average-reward RL. Our key identification is a family of gradient estimators, called score-aware gradient estimators (SAGEs), that enable policy gradient estimation without relying on value-function estimation in the aforementioned setting. We show that SAGE-based policy-gradient locally converges, and we obtain its regret. This includes cases when the state space of the MDP is countable and unstable policies can exist. Under appropriate assumptions such as starting sufficiently close to a maximizer and the existence of a local Lyapunov function, the policy under SAGE-based stochastic gradient ascent has an overwhelming probability of converging to the associated optimal policy. Furthermore, we conduct a numerical comparison between a SAGE-based policy-gradient method and an actor-critic method on several examples inspired from stochastic networks, queueing systems, and models derived from statistical physics. Our results demonstrate that a SAGE-based method finds close-to-optimal policies faster than an actor-critic method.