Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering. However, most previous PbRL work has not investigated the robustness to labeler errors, inevitable with labelers who are non-experts or operate under time constraints. We introduce Similarity as Reward Alignment (SARA), a simple contrastive framework that is both resilient to noisy labels and adaptable to diverse feedback formats. SARA learns a latent representation of preferred samples and computes rewards as similarities to the learned latent. On preference data with varying realistic noise rates, we demonstrate competitive and more stable performance on continuous control offline RL benchmarks, with statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01). We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria. We show that the SARA computed rewards display higher correlation across noise rates compared to baselines.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 60 references that are in the map. First posted on arXiv 14 Jun 2025.
Builds on
45 of 60 references.
Closest ancestors · 14
Same direction, newest first
- 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
- 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learning8 Aug
- 2024Preference Alignment with Flow Matching30 May
- 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
- 2023Nash Learning from Human Feedback1 Dec
- 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
- 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation6 Jun
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
- 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
- 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
- 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
2 more
Other references · 31
- 2017Attention Is All You Needcited by 60,361
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
- 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
19 more
- 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
- 2020dm_control: Software and Tasks for Continuous Controlcited by 375
- 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
- 2020Does label smoothing mitigate label noise?cited by 204
- 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
- 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
- 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
- 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
- 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 89
- 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
- 2020Denoising Implicit Feedback for Recommendationcited by 80
- 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
- 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
- 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
- 2020Hierarchically Decoupled Imitation for Morphological Transfercited by 31
- 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
- 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 15
- 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
- 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Robust Reward Alignment via Hypothesis Space Batch Cutting
- 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
- 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners
- 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
- 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning
- 2024Solving the Inverse Alignment Problem for Efficient RLHF
- 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
- 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling