mlsanityData to 14 Oct 2026

Fine-Tuning Language Models with Advantage-Induced Policy Alignment

arXiv 2306.02231 ↗First posted 4 Jun 2023cs.CL cs.AI cs.LGcited by 48

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as a reliable approach to aligning large language models (LLMs) to human preferences. Among the plethora of RLHF techniques, proximal policy optimization (PPO) is of the most widely used methods. Despite its popularity, however, PPO may suffer from mode collapse, instability, and poor sample efficiency. We show that these issues can be alleviated by a novel algorithm that we refer to as Advantage-Induced Policy Alignment (APA), which leverages a squared error loss function based on the estimated advantages. We demonstrate empirically that APA consistently outperforms PPO in language tasks by a large margin, when a separate reward model is employed as the evaluator. In addition, compared with PPO, APA offers a more stable form of control over the deviation from the model's initial policy, ensuring that the model improves its performance without collapsing to deterministic output. In addition to empirical results, we also provide a theoretical justification supporting the design of our loss function.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 37 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

30 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2015Trust Region Policy Optimizationcited by 2,580
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
16 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  12. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  13. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  14. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  15. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  16. 2022Personalized Reward Learning with Interaction-Grounded Learning (IGL)cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 25; 2025: 7; 2026: 4.

48 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  3. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  5. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  6. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  9. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  10. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  11. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  12. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
36 more
  1. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  2. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  3. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  4. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  5. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  6. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  7. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  8. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  9. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignmentcited by 26
  10. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  11. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 24
  12. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  13. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  14. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  15. 2023Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiencycited by 16
  16. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preferencecited by 12
  17. 2024I am a Strange Dataset: Metalinguistic Tests for Language Modelscited by 10
  18. 2025SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Modelscited by 10
  19. 2024Latent Logic Tree Extraction for Event Sequence Explanation from LLMscited by 9
  20. 2024Reliable Decision Making via Calibration Oriented Retrieval Augmented Generationcited by 6
  21. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  22. 2024Learning to Better Search with Language Models via Guided Reinforced Self-Trainingcited by 5
  23. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 5
  24. 2023ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIFcited by 4
  25. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  26. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  27. 2024Generative AI Security: Challenges and Countermeasurescited by 4
  28. 2024Do LLMs Implicitly Determine the Suitable Text Difficulty for Users?cited by 4
  29. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
  30. 2024Self-Evolution Fine-Tuning for Policy Optimizationcited by 2
  31. 2025Alignment-Aware Decodingcited by 1
  32. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewardscited by 1
  33. 2024Online Self-Preferring Language Modelscited by 0
  34. 2025Thompson Sampling in Online RLHF with General Function Approximationcited by 0
  35. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  36. 2026Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2024Towards Efficient Exact Optimization of Language Model Alignment
  4. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  5. 2025A Principled Loss Function for Direct Language Model Alignment
  6. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  7. 2023Aligning Language Models with Offline Learning from Human Feedback
  8. 2024Accelerated Preference Optimization for Large Language Model Alignment