mlsanityData to 7 Oct 2026

Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult

arXiv 2409.17545 ↗First posted 26 Sep 2024cs.CL cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization methods typically begin training with a well-trained SFT model as a reference model. In RLHF and DPO, a regularization term is used during the preference optimization process to prevent the policy model from deviating too far from the reference model's distribution, thereby avoiding the generation of anomalous responses. When the reference model is already well-aligned with the given data or only requires slight adjustments, this approach can produce a well-aligned model. However, if the reference model is not aligned with the given data and requires significant deviation from its current state, a regularization term may actually hinder the model alignment. In this study, we propose Modulated Intervention Preference Optimization (MIPO) to address this issue. MIPO modulates the degree of intervention from the reference model based on how well the given data is aligned with it. If the data is well-aligned, the intervention is increased to prevent the policy model from diverging significantly from reference model. Conversely, if the alignment is poor, the interference is reduced to facilitate more extensive training. We compare the performance of MIPO and DPO using Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental results demonstrate that MIPO consistently outperforms DPO across various evaluation scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 32 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

24 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Understanding Reference Policies in Direct Preference Optimization18 Jul
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
5 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023Pretraining Language Models with Human Preferencescited by 245
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2024Direct Preference Optimization with an Offset
  4. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization
  5. 2026Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
  6. 2025Why DPO is a Misspecified Estimator and How to Fix It
  7. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  8. 2025What Matters in Data for DPO?