mlsanityData to 7 Oct 2026

AIPO: Improving Training Objective for Iterative Preference Optimization

arXiv 2409.08845 ↗First posted 13 Sep 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at https://github.com/bytedance/AIPO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 34 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

27 of 34 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024sDPO: Don't Use Your Data All at Once28 Mar
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
7 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  3. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  4. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  5. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  6. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  7. 2023Chain of Thought Prompting Elicits Knowledge Augmentationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  3. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  4. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0
  5. 2026Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  2. 2025SGPO: Self-Generated Preference Optimization based on Self-Improver
  3. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  4. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  5. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  6. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  7. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  8. 2024MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples