mlsanityData to 7 Oct 2026

Multi-objective Reinforcement learning from AI Feedback

arXiv 2406.07295 ↗First posted 11 Jun 2024cs.LGcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 27 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

17 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  9. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  10. 2021Alignment of Language Agentscited by 167
  11. 2016Multi-Objective Deep Reinforcement Learningcited by 81
  12. 2017Divide-and-Conquer Reinforcement Learningcited by 46
2 more
  1. 2023Compositional preference models for aligning LMscited by 25
  2. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  2. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 9
  3. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  4. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  5. 2025e-SimFT: Alignment of Generative Models with Simulation Feedback for Pareto-Front Design Explorationcited by 2
  6. 2026Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RLcited by 1
  7. 2026STAGE: Controlled Objective Admission for Multi-Preference LLM Alignmentcited by 1
  8. 2026Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarizationcited by 0
  9. 2026CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  2. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback
  3. 2026Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback
  4. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
  8. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs