mlsanityData to 7 Oct 2026

Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity

arXiv 2505.23923 ↗First posted 29 May 2025cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Currently, most reinforcement learning tasks focus on domains like mathematics and programming, where verification is relatively straightforward. However, in subjective tasks such as role-playing, alignment techniques struggle to make progress, primarily because subjective reward modeling using the Bradley-Terry model faces significant challenges when dealing with ambiguous preferences. To improve reward modeling in subjective tasks, this paper proposes AAM (textbfunderlineAct-textbfunderlineAdaptive textbfunderlineMargin), which enhances reward modeling by dynamically calibrating preference margins using the model's internal parameter knowledge. We design two versions of AAM that efficiently generate contextually-appropriate preference gaps without additional human annotation. This approach fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. To validate AAM's effectiveness in subjective reward modeling, we conduct evaluations on RewardBench, JudgeBench, and challenging role-playing tasks. Results show that AAM significantly improves subjective reward modeling performance, enhancing Bradley-Terry reward models by 2.95% in general tasks and 4.85% in subjective role-playing tasks. Furthermore, reward models trained with AAM can help downstream alignment tasks achieve better results. Our test results show that applying rewards generated by AAM-Augmented RM to preference learning techniques (e.g., GRPO) achieves state-of-the-art results on CharacterEval and Charm. Code and dataset are available at https://github.com/calubkk/AAM.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 42 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

29 of 42 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  2. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives7 Nov
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  4. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  7. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance7 Apr
  8. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  9. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  8. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 139
  11. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  12. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
7 more
  1. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  2. 2024Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversationcited by 55
  3. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  4. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  5. 2025Dialogue Systems for Emotional Support via Value Reinforcementcited by 8
  6. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasetscited by 5
  7. 2024CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimizationcited by 10
  2. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Tool-Augmented Reward Modeling
  2. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
  3. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  4. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
  5. 2026Bayesian Preference Learning for Test-Time Steerable Reward Models
  6. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
  7. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  8. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling