# PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay

arXiv: https://arxiv.org/abs/2112.03798 · HTML: https://mlsanity.com/paper/2112.03798/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Energy-Based Hindsight Experience Prioritization](https://mlsanity.com/paper/1810.01363/) (2018)
- [Adversarial Robustness of Streaming Algorithms through Importance Sampling](https://mlsanity.com/paper/2106.14952/) (2021)
- [Double Prioritized State Recycled Experience Replay](https://mlsanity.com/paper/2007.03961/) (2020)

## Built on by (most cited first, 9 of 9)

- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning](https://mlsanity.com/paper/2503.22456/) (2025)
- [Enhancing PPO with Trajectory-Aware Hybrid Policies](https://mlsanity.com/paper/2502.15968/) (2025)
- [Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization](https://mlsanity.com/paper/2302.02299/) (2023)
- [Evolutionary Enhanced Multi-Agent Reinforcement Learning for Cooperative Air Combat](https://mlsanity.com/paper/2605.25091/) (2026)
- [Reputation-Driven Asynchronous Federated Learning for Enhanced Trajectory Prediction with Blockchain](https://mlsanity.com/paper/2407.19428/) (2024)
- [Not All Transitions Matter: Evidence from PPO](https://mlsanity.com/paper/2605.24071/) (2026)
- [Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF](https://mlsanity.com/paper/2607.07693/) (2026)
- [Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?](https://mlsanity.com/paper/2610.01399/) (2026)

## Abstract (arXiv)

On-policy deep reinforcement learning algorithms have low data utilization and require significant experience for policy improvement. This paper proposes a proximal policy optimization algorithm with prioritized trajectory replay (PTR-PPO) that combines on-policy and off-policy methods to improve sampling efficiency by prioritizing the replay of trajectories generated by old policies. We first design three trajectory priorities based on the characteristics of trajectories: the first two being max and mean trajectory priorities based on one-step empirical generalized advantage estimation (GAE) values and the last being reward trajectory priorities based on normalized undiscounted cumulative reward. Then, we incorporate the prioritized trajectory replay into the PPO algorithm, propose a truncated importance weight method to overcome the high variance caused by large importance weights under multistep experience, and design a policy improvement loss function for PPO under off-policy conditions. We evaluate the performance of PTR-PPO in a set of Atari discrete control tasks, achieving state-of-the-art performance. In addition, by analyzing the heatmap of priority changes at various locations in the priority memory during training, we find that memory size and rollout length can have a significant impact on the distribution of trajectory priorities and, hence, on the performance of the algorithm.