# Proximal Policy Optimization and its Dynamic Version for Sequence Generation

arXiv: https://arxiv.org/abs/1808.07982 · HTML: https://mlsanity.com/paper/1808.07982/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2018.

## Built on by (most cited first, 7 of 7)

- [Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models](https://mlsanity.com/paper/2404.01295/) (2024)
- [Towards Cost-Effective Reward Guided Text Generation](https://mlsanity.com/paper/2502.04517/) (2025)
- [Generate and Revise: Reinforcement Learning in Neural Poetry](https://mlsanity.com/paper/2102.04114/) (2021)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Enhancing Quantum Memory Lifetime with Measurement-Free Local Error Correction and Reinforcement Learning](https://mlsanity.com/paper/2408.09524/) (2024)
- [A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models](https://mlsanity.com/paper/2408.16751/) (2024)
- [Trait-Aware Policy Optimization for Autoregressive Multi-Trait Essay Scoring](https://mlsanity.com/paper/2605.25731/) (2026)

## Abstract (arXiv)

In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance.