mlsanityData to 14 Oct 2026

REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective

arXiv 2504.11337 ↗First posted 15 Apr 2025cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Multi-objective preference alignment in language models often encounters a challenging trade-off: optimizing for one human preference (e.g., helpfulness) frequently compromises others (e.g., harmlessness) due to the inherent conflicts between competing objectives. While prior work mainly focuses on algorithmic solutions, we explore a novel data-driven approach to uncover the types of data that can effectively mitigate these conflicts. Specifically, we propose the concept of Reward Consistency (RC), which identifies samples that align with multiple preference objectives, thereby reducing conflicts during training. Through gradient-based analysis, we demonstrate that RC-compliant samples inherently constrain performance degradation during multi-objective optimization. Building on these insights, we further develop Reward Consistency Sampling, a framework that automatically constructs preference datasets that effectively mitigate conflicts during multi-objective alignment. Our generated data achieves an average improvement of 13.37% in both the harmless rate and helpfulness win rate when optimizing harmlessness and helpfulness, and can consistently resolve conflicts in varying multi-objective scenarios.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 39 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

28 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins12 Oct
  2. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization5 Sep
  3. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling21 May
  6. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  9. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  10. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  11. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
3 more
  1. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  12. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
1 more
  1. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  2. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  3. 2026STAGE: Controlled Objective Admission for Multi-Preference LLM Alignmentcited by 1
  4. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 0
  5. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-free Alignment for Conflicting Objectives
  2. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  3. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  4. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  5. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  6. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
  7. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  8. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs