mlsanityData to 7 Oct 2026

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

arXiv 2311.08045 ↗First posted 14 Nov 2023cs.CL cs.AI cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Human preference alignment is essential to improve the interaction quality of large language models (LLMs). Existing alignment methods depend on manually annotated preference data to guide the LLM optimization directions. However, continuously updating LLMs for alignment raises a distribution gap between model-generated samples and human-annotated responses, hindering training effectiveness. To mitigate this issue, previous methods require additional preference annotation on newly generated samples to adapt to the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a min-max game. Through adversarial training, the reward model can adapt to the shifted generation distribution of the LLM without any additional annotation. With comprehensive experiments, we find the proposed adversarial training framework further enhances existing alignment baselines in terms of LLM helpfulness and harmlessness. The code is at https://github.com/Linear95/APO.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 49 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2014Conditional Generative Adversarial Netscited by 2,152
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
15 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2023Instruction Tuning with GPT-4cited by 676
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  7. 2023Mathematical Capabilities of ChatGPTcited by 263
  8. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  9. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  10. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  11. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  12. 2017Adversarial Feature Matching for Text Generationcited by 58
  13. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46
  14. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  15. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 19; 2025: 6; 2026: 1.

27 papers.

Sort:

By area:

  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  4. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  5. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  6. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  7. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  8. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 33
  9. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  10. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  11. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  12. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
15 more
  1. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  2. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  3. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
  4. 2024Boost Your Human Image Generation Model via Direct Preference Optimizationcited by 16
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  6. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  7. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguardscited by 14
  8. 2024Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generationcited by 9
  9. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  10. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  11. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  12. 2025DAP: A Discrete-token Autoregressive Planner for Autonomous Drivingcited by 4
  13. 2024Holistic Utility Preference Learning for Listwise Alignmentcited by 2
  14. 2025SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generationcited by 0
  15. 2026Latent Adversarial Regularization for Offline Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Preference Learning for Robust LLM Alignment
  2. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  4. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  5. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2023Preference Ranking Optimization for Human Alignment
  8. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts