mlsanityData to 14 Oct 2026

Exploring and Addressing Reward Confusion in Offline Preference Learning

arXiv 2407.16025 ↗First posted 22 Jul 2024cs.LG cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 23 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

12 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018DeepMind Control Suitecited by 956
  3. 2019Causal Confusion in Imitation Learningcited by 288
  4. 2022Defining and Characterizing Reward Hackingcited by 203
  5. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  6. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  7. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  8. 2022Causal Imitation Learning with Unobserved Confounderscited by 41
  9. 2022Sequential Causal Imitation Learning with Unobserved Confounderscited by 25
  10. 2020Causal World Models by Unsupervised Deconfounding of Physical Dynamicscited by 15
  11. 2018Reward Learning from Narrated Demonstrationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  3. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
  4. 2024Preference Elicitation for Offline Reinforcement Learning
  5. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  6. 2021Offline Preference-Based Apprenticeship Learning
  7. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective