# Ranking Policy Gradient

arXiv: https://arxiv.org/abs/1906.09674 · HTML: https://mlsanity.com/paper/1906.09674/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2019.

## Built on by (most cited first, 4 of 4)

- [Regularized Behavior Value Estimation](https://mlsanity.com/paper/2103.09575/) (2021)
- [Simplifying Deep Reinforcement Learning via Self-Supervision](https://mlsanity.com/paper/2106.05526/) (2021)
- [SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin](https://mlsanity.com/paper/2502.13516/) (2025)
- [Age of Semantics in Cooperative Communications: To Expedite Simulation Towards Real via Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08947/) (2022)

## Abstract (arXiv)

Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency. Furthermore, the sample complexity of RPG does not depend on the dimension of state space, which enables RPG for large-scale problems. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.