mlsanityData to 14 Oct 2026

RVPO: Risk-Sensitive Alignment via Variance Regularization

arXiv 2605.05750 ↗First posted 7 May 2026cs.LG cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing "bottleneck" rewards vital for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation, shifting the objective from "maximize sum" to "maximize consistency." We show via Taylor expansion that a LogSumExp (SoftMin) operator effectively acts as a smooth variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 concurrent LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). By preventing the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench (0.261 vs. 0.215 for GDPO at 14B, p < 0.001) and maintains competitive accuracy on GPQA-Diamond without the late-stage degradation observed in other multi-reward methods, demonstrating that variance regularization mitigates constraint neglect across model scales without sacrificing general capabilities.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 39 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

33 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Reward-free Alignment for Conflicting Objectives2 Feb
  2. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework5 Jun
  3. 2025Multi-objective Large Language Model Alignment with Hierarchical Experts27 May
  4. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  10. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  12. 2025ToolRL: Reward is All Tool Learning Needscited by 362
14 more
  1. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  2. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  4. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  5. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  7. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  8. 2024Rule Based Rewards for Language Model Safetycited by 108
  9. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  10. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
  11. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  12. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  13. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  14. 2020Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 4
  2. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  3. 2026Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
  2. 2026f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment
  3. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
  4. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
  5. 2026ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients
  6. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
  7. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
  8. 2025Preference Robustness for DPO with Applications to Public Health