mlsanityData to 14 Oct 2026

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

arXiv 2607.29246 ↗First posted 31 Jul 2026cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 46 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

37 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024Multi-objective Reinforcement learning from AI Feedback11 Jun
  3. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling2 Mar
  4. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2017Constrained Policy Optimizationcited by 983
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2025Group Sequence Policy Optimizationcited by 835
17 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2022Efficiently Scaling Transformer Inferencecited by 539
  4. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  5. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  6. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  8. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  9. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  10. 2023Mitigating the Alignment Tax of RLHFcited by 183
  11. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  12. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  13. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 74
  14. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  15. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 23
  16. 2024Dynamic Multi-Reward Weighting for Multi-Style Controllable Generationcited by 13
  17. 2025Dichotomous Diffusion Policy Optimizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models
  2. 2024The Perfect Blend: Redefining RLHF with Mixture of Judges
  3. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  4. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models
  5. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
  6. 2024Fine-Tuning Language Models with Reward Learning on Policy
  7. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  8. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization