mlsanityData to 13 Oct 2026

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

arXiv 2503.10669 ↗First posted 10 Mar 2025cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 42 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

33 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  2. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  4. 2017Deep reinforcement learning from human preferencescited by 3,871
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  6. 2020Learning to summarize from human feedbackcited by 2,575
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 745
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 680
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  10. 2016Input Convex Neural Networkscited by 408
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 371
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 347
17 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 240
  2. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 232
  3. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  4. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 185
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  6. 2024Personalization of Large Language Models: A Surveycited by 104
  7. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 88
  8. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  9. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  10. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 79
  11. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  12. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 43
  13. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 42
  14. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 34
  15. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  16. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  17. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 22
  2. 2025ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Codecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  2. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
  3. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  4. 2026One Model for All: Multi-Objective Controllable Language Models
  5. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  6. 2026Reward-free Alignment for Conflicting Objectives
  7. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  8. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment