Robust Reward Alignment via Hypothesis Space Batch Cutting
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Reward design in reinforcement learning and optimal control is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in a model predictive control setting across diverse tasks. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 57 references that are in the map. First posted on arXiv 5 Feb 2025.
Builds on
38 of 57 references.
Closest ancestors · 12
Same direction, newest first
- 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback4 Feb
- 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
- 2023Active Reward Learning from Online Preferences27 Feb
- 2023Reinforcement Learning from Diverse Human Preferences27 Jan
- 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
- 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
- 2021Learning Multimodal Rewards from Rankings27 Sep
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
- 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
- 2018Reward learning from human preferences and demonstrations in Atari15 Nov
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 26
- 2023GPT-4 Technical Reportcited by 18,132
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
- 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
- 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
- 2020Does label smoothing mitigate label noise?cited by 204
- 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
- 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
- 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 132
- 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
- 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
14 more
- 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 105
- 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
- 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
- 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
- 2019Robust Inference via Generative Classifiers for Handling Noisy Labelscited by 67
- 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
- 2017Deep Learning from Noisy Image Labels with Quality Embeddingcited by 46
- 2019DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learningcited by 45
- 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 39
- 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
- 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
- 2020Learning from Human Directional Correctionscited by 5
- 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedbackcited by 3
- 2024Safe MPC Alignment with Human Directional Feedbackcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
- 2024Towards Reliable Alignment: Uncertainty-aware RLHF
- 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
- 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
- 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
- 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
- 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
- 2023REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback