# Exploring and Addressing Reward Confusion in Offline Preference Learning

arXiv: https://arxiv.org/abs/2407.16025 · HTML: https://mlsanity.com/paper/2407.16025/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Causal Imitation Learning with Unobserved Confounders](https://mlsanity.com/paper/2208.06267/) (2022)
- [Sequential Causal Imitation Learning with Unobserved Confounders](https://mlsanity.com/paper/2208.06276/) (2022)
- [Causal World Models by Unsupervised Deconfounding of Physical Dynamics](https://mlsanity.com/paper/2012.14228/) (2020)
- [Reward Learning from Narrated Demonstrations](https://mlsanity.com/paper/1804.10692/) (2018)

## Built on by (most cited first, 1 of 1)

- [Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders](https://mlsanity.com/paper/2605.16339/) (2026)

## Abstract (arXiv)

Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.