mlsanityData to 14 Oct 2026

Evaluating Feature Dependent Noise in Preference-based Reinforcement Learning

arXiv 2601.01904 ↗First posted 5 Jan 2026cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Learning from Preferences in Reinforcement Learning (PbRL) has gained attention recently, as it serves as a natural fit for complicated tasks where the reward function is not easily available. However, preferences often come with uncertainty and noise if they are not from perfect teachers. Much prior literature aimed to detect noise, but with limited types of noise and most being uniformly distributed with no connection to observations. In this work, we formalize the notion of targeted feature-dependent noise and propose several variants like trajectory feature noise, trajectory similarity noise, margin dependent noise, and Language Model noise. We evaluate feature-dependent noise, where noise is correlated with certain features in complex continuous control tasks from DMControl and Meta-world. Our experiments show that in some feature-dependent noise settings, the state-of-the-art noise-robust PbRL method's learning performance is significantly deteriorated, while PbRL method with no explicit denoising can surprisingly outperform noise-robust PbRL in the majority of settings. We also find language models' noise exhibits similar characteristics to feature-dependent noise, thereby simulating realistic humans and call for further study in learning with feature-dependent noise robustly.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 36 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

24 of 36 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning14 Jun
  2. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations9 May
  3. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model22 Dec
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023Reinforcement Learning from Diverse Human Preferences27 Jan
  6. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  7. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  5. 2018DeepMind Control Suitecited by 956
  6. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  7. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  9. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2021Reward (Mis)design for Autonomous Drivingcited by 104
4 more
  1. 2021Instance-dependent Label-noise Learning under a Structural Causal Modelcited by 50
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  3. 2023GVFs in the Real World: Making Predictions Online for Water Treatmentcited by 13
  4. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 13
  2. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferences
  2. 2026Optimal Transport for LLM Reward Modeling from Noisy Preference
  3. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
  4. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback
  5. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?
  6. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
  7. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective