mlsanityData to 14 Oct 2026

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

arXiv 2310.00212 ↗First posted 30 Sep 2023cs.LG cs.AI cs.CLcited by 57

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as the default RL optimizer. Despite its effectiveness, PPO has limitations when optimizing rewards trained from comparison-based loss. Primarily, PPO is not invariant to equivalent reward functions containing identical preference information due to the need to calibrate the reward scale. Additionally, PPO's necessity for token-wise updates introduces complexity in both function approximation and algorithm design compared to trajectory-wise optimization. This paper proposes a new framework, reinforcement learning with relative feedback, and a novel trajectory-wise policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O) that operates directly on comparative rewards. We show theoretically that P3O is invariant to equivalent rewards and avoids the complexity of PPO. Empirical evaluations demonstrate that P3O outperforms PPO in the KL-Reward trade-off and can align with human preferences as well as or better than prior methods. In summary, this work introduces a simpler yet effective approach for aligning LLMs to human preferences through relative feedback.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

36 of 47 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  3. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
13 more
  1. 2022Red Teaming Language Models with Language Modelscited by 1,026
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  3. 2023Textbooks Are All You Needcited by 569
  4. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  6. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  7. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  8. 2015Contextual Dueling Banditscited by 101
  9. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  10. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  11. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  12. 2021Nearly Optimal Policy Optimization with Stable at Any Time Guaranteecited by 13
  13. 2021A Reduction-Based Framework for Conservative Bandits and Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 31; 2025: 16; 2026: 5.

57 papers.

Sort:

By area:

  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  3. 2024Generative Reward Modelscited by 148
  4. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  6. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  7. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  8. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  9. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  10. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  11. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  12. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
38 more
  1. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  2. 2025Reinforcement Learning from Human Feedbackcited by 58
  3. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  4. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  5. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  6. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  7. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  8. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
  9. 2024Large Language Models for Education: A Surveycited by 34
  10. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 30
  11. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  12. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  13. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  14. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  15. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  16. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  17. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  18. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  19. 2025CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIscited by 14
  20. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  21. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  22. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 12
  23. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 12
  24. 2024H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMscited by 11
  25. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  26. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  27. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferencescited by 10
  28. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularizationcited by 9
  29. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  30. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  31. 2025Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learningcited by 7
  32. 2024Active Preference Learning for Ordering Items In- and Out-of-samplecited by 6
  33. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
  34. 2023An Invitation to Deep Reinforcement Learningcited by 5
  35. 2024TODO: Enhancing LLM Alignment with Ternary Preferencescited by 5
  36. 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
  37. 2023COPR: Continual Learning Human Preference through Optimal Policy Regularizationcited by 2
  38. 2025GradEscape: A Gradient-Based Evader Against AI-Generated Text Detectorscited by 2

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  2. 2024Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  4. 2024Accelerated Preference Optimization for Large Language Model Alignment
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
  7. 2024Towards Efficient Exact Optimization of Language Model Alignment
  8. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models