mlsanityData to 14 Oct 2026

Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length

arXiv 2308.05585 ↗First posted 10 Aug 2023cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The Reinforcement Learning from Human Feedback (RLHF) plays a pivotal role in shaping the impact of large language models (LLMs), contributing significantly to controlling output toxicity and selecting output styles, particularly as LLMs often harbor misleading content, highlighting the urgency to align them with human values for secure AI systems. The RLHF, characterized by complexity, instability, and sensitivity to hyperparameters, makes the evaluation of the reward model for complex tasks challenging, thereby further complicating the use of Proximal Policy Optimization (PPO). In this paper, we introduce a simple task designed to employ Gloden as a reward model that validates the effectiveness of PPO and inspires it, primarily explaining the task of utilizing PPO to manipulate the tokenizer length of the output generated by the model. Experiments confirm that PPO is not only effective in manipulating the output tokenizer length to a certain extent in this type of task but also exhibits facilitated training once the influence of the reward model effect is excluded, making it an exciting development.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 27 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

18 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  8. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  9. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  10. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  11. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  12. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 106
3 more
  1. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
  2. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialogcited by 37
  3. 2019Conservative Agency via Attainable Utility Preservationcited by 35

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Secrets of RLHF in Large Language Models Part I: PPO
  2. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF
  3. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models
  4. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  5. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
  6. 2023Efficient RLHF: Reducing the Memory Usage of PPO
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  8. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment