mlsanityData to 7 Oct 2026

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

arXiv 2606.07705 ↗First posted 5 Jun 2026cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Although multi-objective reinforcement learning (MORL) is central to aligning large language models with complex human preferences, the prevailing practice of static weighted summation overlooks a more fundamental phenomenon: reward learning is markedly asynchronous across objectives. Well-learned dimensions quickly produce homogeneous, low-variance signals whose residual noise contaminates the aggregated reward (in GRPO) or occupies a fixed share of the advantage budget (in GDPO), interfering with the scarce yet high-value signals carried by under-learned dimensions. To address this asynchrony, we propose Stage-Aware Dynamic Weighting (SAW), a lightweight, algorithm-agnostic dynamic weighting mechanism. SAW utilizes the coefficient of variation (CV) as a scale-invariant proxy for real-time informativeness, reweighting each dimension's reward or advantage contribution by its relative informativeness within the batch. Unlike gradient-based methods that require multiple forward and backward passes, SAW relies solely on batch-level statistics, introducing nearly negligible computational overhead. Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance under both GRPO and GDPO frameworks, confirming it as a general-purpose plug-in for multi-reward LLM alignment. Our code is available at https://github.com/Zhaolutuan/SAW

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 36 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

29 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  2. 2024ALaRM: Align Language Models via Hierarchical Rewards Modeling11 Mar
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  10. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
12 more
  1. 2025Training Language Models to Reason Efficientlycited by 242
  2. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  3. 2025Reinforcement Learning with Rubric Anchorscited by 108
  4. 2024Rule Based Rewards for Language Model Safetycited by 106
  5. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  6. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  7. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  8. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  9. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  10. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  11. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  12. 2026Reward-free Alignment for Conflicting Objectivescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2
  2. 2026MatrixReward: Reward from Rubric Matrix for Open-Ended Generationcited by 0
  3. 2026Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies
  2. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  3. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  4. 2025Dynamic Policy Induction for Adaptive Prompt Optimization: Bridging the Efficiency-Accuracy Gap via Lightweight Reinforcement Learning
  5. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  7. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
  8. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning