mlsanityData to 7 Oct 2026

Preference Learning for AI Alignment: a Causal Perspective

arXiv 2506.05967 ↗First posted 6 Jun 2025cs.AI cs.LG stat.MLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward modelling from preference data is a crucial step in aligning large language models (LLMs) with human values, requiring robust generalisation to novel prompt-response pairs. In this work, we propose to frame this problem in a causal paradigm, providing the rich toolbox of causality to identify the persistent challenges, such as causal misidentification, preference heterogeneity, and confounding due to user-specific factors. Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation and contrast them with common data collection practices. We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness. Finally, we outline desiderata for future research and practices, advocating targeted interventions to address inherent limitations of observational data.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 62 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

44 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals15 Oct
  2. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  3. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  5. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  9. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  11. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2019Causal Confusion in Imitation Learningcited by 286
  6. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  7. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 154
  8. 2022Weakly supervised causal representation learningcited by 140
  9. 2022Interventional Causal Representation Learningcited by 140
  10. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  11. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  12. 2024Robust agents learn causal world modelscited by 81
20 more
  1. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  2. 2021Causally motivated Shortcut Removal Using Auxiliary Labelscited by 55
  3. 2024Active Preference Learning for Large Language Modelscited by 55
  4. 2024Controllable Text Generation for Large Language Models: A Surveycited by 55
  5. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  6. 2022A Causal Lens for Controllable Text Generationcited by 45
  7. 2020Causal Effects of Linguistic Propertiescited by 39
  8. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  9. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  10. 2017Estimation of causal effects with multiple treatments: a review and new ideascited by 29
  11. 2019Adversarial Balancing-based Representation Learning for Causal Effect Inference with Observational Datacited by 29
  12. 2024LLMs + Persona-Plug = Personalized LLMscited by 29
  13. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  14. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  15. 2018Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processingcited by 15
  16. 2022Causal Estimation for Text Data with (Apparent) Overlap Violationscited by 15
  17. 2024Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environmentcited by 12
  18. 2018Adversarial Balancing for Causal Inferencecited by 10
  19. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problemcited by 8
  20. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Interpretable Reward Modeling with Active Concept Bottleneckscited by 3
  2. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  3. 2026Causal methods for LLM development and evaluationcited by 2
  4. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialoguecited by 1
  5. 2026PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibrationcited by 0
  6. 2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHFcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment is Localized: A Causal Probe into Preference Layers
  2. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
  3. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  4. 2024Transforming and Combining Rewards for Aligning Large Language Models
  5. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  6. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  7. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback