mlsanityData to 7 Oct 2026

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

arXiv 2507.21183 ↗First posted 27 Jul 2025cs.LG cs.AI cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 72 references that are in the map. First posted on arXiv 27 Jul 2025.

Builds on

53 of 72 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO29 May
  2. 2024Weighted-Reward Preference Optimization for Implicit Model Fusion4 Dec
  3. 2024TODO: Enhancing LLM Alignment with Ternary Preferences2 Nov
  4. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  5. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  6. 2024Learning Dynamics of LLM Finetuning15 Jul
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  10. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  11. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
7 more
  1. 2024Direct Preference Optimization with an Offset16 Feb
  2. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2023Mitigating the Alignment Tax of RLHF12 Sep
  6. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2023Mistral 7Bcited by 3,065
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
22 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  5. 2025Group Sequence Policy Optimizationcited by 788
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  13. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  14. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  15. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  16. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  17. 2024Nemotron-4 340B Technical Reportcited by 143
  18. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  19. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  20. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  21. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 30
  22. 2024Reward Learning From Preference With Tiescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  2. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  3. 2025Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoningcited by 15
  4. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  5. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 6
  6. 2026Recursive Self-Evolving Agents via Held-Out Selectioncited by 6
  7. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  8. 2026MIS-HCC: Hierarchical Channel Clustering for Efficient Medical Image Segmentationcited by 4
  9. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  10. 2026Local Epistemic Uncertainty Guided Active Sampling for Plug-and-play Diffusive Image Restorationcited by 2
  11. 2026"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?cited by 1
  12. 2025Quality analysis and evaluation prediction of RAG retrieval based on machine learning algorithmscited by 0
2 more
  1. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0
  2. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
  2. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  3. 2025A Survey of Direct Preference Optimization
  4. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  5. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
  6. 2024Multi-Reference Preference Optimization for Large Language Models
  7. 2026Autoregressive Direct Preference Optimization
  8. 2025SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models