mlsanityData to 7 Oct 2026

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

arXiv 2501.09620 ↗First posted 16 Jan 2025cs.LG cs.AIcited by 42

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated significant progress in performing complex tasks. While Reinforcement Learning from Human Feedback (RLHF) has been effective in aligning LLMs with human preferences, it is susceptible to spurious correlations in reward modeling. Consequently, it often introduces biases-such as length bias, sycophancy, conceptual bias, and discrimination-that hinder the model's ability to capture true causal relationships. To address this, we propose a novel causal reward modeling approach that integrates causality to mitigate these spurious correlations. Our method enforces counterfactual invariance, ensuring reward predictions remain consistent when irrelevant variables are altered. Through experiments on both synthetic and real-world datasets, we show that our approach mitigates various types of spurious correlations effectively, resulting in more reliable and fair alignment of LLMs with human preferences. As a drop-in enhancement to the existing RLHF workflow, our causal reward modeling provides a practical way to improve the trustworthiness and fairness of LLM finetuning.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 66 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

57 of 66 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024GRAPE: Generalizing Robot Policy via Preference Alignment28 Nov
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Bias25 Sep
  3. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  4. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  5. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  7. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  8. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  9. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  10. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
2 more
  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  12. 2016Concrete Problems in AI Safetycited by 1,582
31 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  8. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  9. 2018Maximum a Posteriori Policy Optimisationcited by 318
  10. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  13. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  14. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  15. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 129
  16. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  17. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  18. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  19. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 61
  20. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  21. 2023The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendationscited by 41
  22. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  23. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  24. 2024Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancycited by 20
  25. 2024BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimizationcited by 18
  26. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  27. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  28. 2016Length bias in Encoder Decoder Models and a Case for Global Conditioningcited by 14
  29. 2022Learning Counterfactually Invariant Predictorscited by 10
  30. 2022Selection Bias Induced Spurious Correlations in Large Language Modelscited by 9
  31. 2023Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguardscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 21.

42 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  4. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  5. 2025Supervising the search process produces reliable and generalizable information-seeking agentscited by 20
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  7. 2025Robust Reward Modeling via Causal Rubricscited by 18
  8. 2025Virtual Agent Economiescited by 17
  9. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  10. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 9
  11. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  12. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
30 more
  1. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 8
  2. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  3. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  4. 2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
  5. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 6
  6. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  7. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  8. 2025Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimizationcited by 4
  9. 2025Mitigating Length Bias in RLHF through a Causal Lenscited by 4
  10. 2025S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuningcited by 3
  11. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  12. 2026SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimizationcited by 3
  13. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3
  14. 2025Causally-Enhanced Reinforcement Policy Optimizationcited by 2
  15. 2025Learning "Partner-Aware" Collaborators in Multi-Party Collaborationcited by 2
  16. 2026Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Rewardcited by 2
  17. 2026DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivitycited by 2
  18. 2025Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspectivecited by 1
  19. 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignmentcited by 1
  20. 2026Factored Causal Representation Learning for Robust Reward Modeling in RLHFcited by 1
  21. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  22. 2026General Preference Reinforcement Learningcited by 1
  23. 2025Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinementcited by 0
  24. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  25. 2026Robust Reward Modeling for Large Language Models via Causal Decompositioncited by 0
  26. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 0
  27. 2026Causality as the Statistical Conscience of Artificial Intelligence: From Pearl's Ladder to Trustworthy Machinescited by 0
  28. 2026Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluationcited by 0
  29. 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Modelscited by 0
  30. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Learning for AI Alignment: a Causal Perspective
  2. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking
  3. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
  4. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  5. 2026Debiasing Reward Models via Causally Motivated Inference-Time Intervention
  6. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations
  7. 2025Alignment is Localized: A Causal Probe into Preference Layers
  8. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment