mlsanityData to 7 Oct 2026

Causal Confusion and Reward Misidentification in Preference-Based Reward Learning

arXiv 2204.06601 ↗First posted 13 Apr 2022cs.LG cs.ROcited by 52

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning policies via preference-based reward learning is an increasingly popular method for customizing agent behavior, but has been shown anecdotally to be prone to spurious correlations and reward hacking behaviors. While much prior work focuses on causal confusion in reinforcement learning and behavioral cloning, we focus on a systematic study of causal confusion and reward misidentification when learning from preferences. In particular, we perform a series of sensitivity and ablation analyses on several benchmark domains where rewards learned from preferences achieve minimal test error but fail to generalize to out-of-distribution states -- resulting in poor policy performance when optimized. We find that the presence of non-causal distractor features, noise in the stated preferences, and partial state observability can all exacerbate reward misidentification. We also identify a set of methods with which to interpret misidentified learned rewards. In general, we observe that optimizing misidentified rewards drives the policy off the reward's training distribution, resulting in high predicted (learned) rewards but low true rewards. These findings illuminate the susceptibility of preference learning to reward misidentification and causal confusion -- failure to consider even one of many factors can result in unexpected, undesirable behavior.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 38 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

23 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2018Behavioral Cloning from Observationcited by 482
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  7. 2019Causal Confusion in Imitation Learningcited by 286
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  9. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  12. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
9 more
  1. 2020Quantifying Differences in Reward Functionscited by 62
  2. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  3. 2017Variational Inference using Implicit Distributionscited by 39
  4. 2020Understanding Learned Reward Functionscited by 37
  5. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  6. 2020Feature Expansive Reward Learning: Rethinking Human Inputcited by 25
  7. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  8. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
  9. 2021Assisted Robust Reward Designcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 20; 2025: 11; 2026: 8.

52 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  7. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  8. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  9. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  10. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  11. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  12. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
38 more
  1. 2024Fine-Tuning Language Models with Reward Learning on Policycited by 15
  2. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  3. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  5. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  6. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  7. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  8. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  9. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  10. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  11. 2024PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learningcited by 8
  12. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  13. 2025Multiplayer Nash Preference Optimizationcited by 7
  14. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  15. 2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
  16. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learningcited by 5
  17. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  18. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  19. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  20. 2024Dishonesty in Helpful and Harmless Alignmentcited by 3
  21. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3
  22. 2024Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocauterycited by 2
  23. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2
  24. 2026ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injectioncited by 2
  25. 2023Quantifying Assistive Robustness Via the Natural-Adversarial Frontiercited by 1
  26. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?cited by 1
  27. 2025Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspectivecited by 1
  28. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0
  29. 2023Contextual Reliability: When Different Features Matter in Different Contextscited by 0
  30. 2024Value Internalization: Learning and Generalizing from Social Rewardcited by 0
  31. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
  32. 2024Choice Between Partial Trajectories: Disentangling Goals from Beliefscited by 0
  33. 2024Adaptive Querying for Reward Learning from Human Feedbackcited by 0
  34. 2025Aligning LLMs with Domain Invariant Reward Modelscited by 0
  35. 2025Dataset Poisoning Attacks on Behavioral Cloning Policiescited by 0
  36. 2026Prefer-DAS: Learning from Local Preferences and Sparse Prompts for Domain Adaptive Segmentation of Electron Microscopycited by 0
  37. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 0
  38. 2026Implicit Safety Alignment from Crowd Preferencescited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring and Addressing Reward Confusion in Offline Preference Learning
  2. 2023On The Fragility of Learned Reward Functions
  3. 2026A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR
  4. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  5. 2021Exploration and preference satisfaction trade-off in reward-free learning
  6. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  7. 2026An Introduction to Causal Reinforcement Learning
  8. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences