# Adaptive Preference Scaling for Reinforcement Learning with Human Feedback arXiv: https://arxiv.org/abs/2406.02764 ยท HTML: https://mlsanity.com/paper/2406.02764/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. - Builds on 33 of its 45 references that are in the map. - First posted on arXiv 4 Jun 2024. ## Placed in - LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/) ## Builds on (most cited first, 30 of 33) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020) - [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016) - [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019) - [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022) - [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023) - [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023) - [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023) - [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020) - [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021) - [Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach](https://mlsanity.com/paper/1610.03425/) (2016) - [Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning](https://mlsanity.com/paper/1908.08729/) (2019) - [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019) - [SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2203.10050/) (2022) - [Local Temperature Scaling for Probability Calibration](https://mlsanity.com/paper/2008.05105/) (2020) - [Data-Driven Robust Optimization](https://mlsanity.com/paper/1401.0212/) (2013) - [Smooth Exploration for Robotic Reinforcement Learning](https://mlsanity.com/paper/2005.05719/) (2020) - [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022) - [PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2308.02585/) (2023) - [Modeling the Second Player in Distributionally Robust Optimization](https://mlsanity.com/paper/2103.10282/) (2021) - [Contextual Temperature for Language Modeling](https://mlsanity.com/paper/2012.13575/) (2020) - [Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networks](https://mlsanity.com/paper/2208.00461/) (2022) - [Reward Collapse in Aligning Large Language Models](https://mlsanity.com/paper/2305.17608/) (2023) ## Built on by (most cited first, 20 of 20) - [Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison](https://mlsanity.com/paper/2511.07919/) (2025) - [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025) - [Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models](https://mlsanity.com/paper/2505.16265/) (2025) - [Pairwise Calibrated Rewards for Pluralistic Alignment](https://mlsanity.com/paper/2506.06298/) (2025) - [Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning](https://mlsanity.com/paper/2504.03784/) (2025) - [Distributionally Robust Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2503.00539/) (2025) - [Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning](https://mlsanity.com/paper/2604.12479/) (2026) - [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025) - [Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2505.21907/) (2025) - [Lightweight Robust Direct Preference Optimization](https://mlsanity.com/paper/2510.23590/) (2025) - [AI Alignment From Social Choice Perspectives](https://mlsanity.com/paper/2606.21550/) (2026) - [MixDPO: Modeling Preference Strength for Pluralistic Alignment](https://mlsanity.com/paper/2601.06180/) (2026) - [Mitigating loss of control in advanced AI systems through instrumental goal trajectories](https://mlsanity.com/paper/2602.01699/) (2026) - [DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection](https://mlsanity.com/paper/2508.18474/) (2025) - [Performance Assessment Strategies for Language Model Applications in Healthcare](https://mlsanity.com/paper/2509.08087/) (2025) - [Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approach](https://mlsanity.com/paper/2511.12351/) (2025) - [From Search to Synthesis: Training LLMs as Zero-Shot Workflow Generators](https://mlsanity.com/paper/2606.30704/) (2026) - [Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation](https://mlsanity.com/paper/2608.08494/) (2026) - [Uncertainty-Normalized Margins for Direct Preference Optimization](https://mlsanity.com/paper/2609.38647/) (2026) - [Robust Nash Alignment under Preference Uncertainty](https://mlsanity.com/paper/2610.00715/) (2026) ## Abstract (arXiv) Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.