mlsanityData to 14 Oct 2026

Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

arXiv 2502.14354 ↗First posted 20 Feb 2025cs.LG cs.CLcited by 16

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines. Code is available at https://github.com/zyttt-coder/SIPO.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 38 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

25 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  7. 2022Large Language Models Can Self-Improvecited by 664
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  12. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
10 more
  1. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  2. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  3. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  5. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  6. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  7. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  8. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  9. 2024CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAGcited by 22
  10. 2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 45
  3. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 23
  4. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  5. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  6. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  7. 2025From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generationcited by 4
  8. 2026AI Alignment Breaks at the Edgecited by 4
  9. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  10. 2026Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMscited by 1
  11. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  12. 2026LLM Output Detectability and Task Performance Can be Jointly Optimizedcited by 0
4 more
  1. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  2. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  3. 2026Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignmentcited by 0
  4. 2026Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective
  2. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  3. 2025Robust Multi-Objective Preference Alignment with Online DPO
  4. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  5. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences
  6. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  7. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework
  8. 2025Rethinking DPO: The Role of Rejected Responses in Preference Misalignment