mlsanityData to 7 Oct 2026

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals

arXiv 2410.11348 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper we develop Rewrite-based Attribute Treatment Estimator (RATE) as an effective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Importantly, RATE measures the causal effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfactuals examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substantial bias in the estimated sensitivity of the reward model to the attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting twice. We establish the validity of the RATE procedure and show empirically that it is an effective estimator.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 27 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

13 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr

Other references · 12

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  6. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  7. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  8. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  9. 2022Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lenscited by 28
  10. 2024Zero-shot LLM-guided Counterfactual Generation: A Case Study on NLP Model Evaluationcited by 17
  11. 2022Causal Estimation for Text Data with (Apparent) Overlap Violationscited by 15
  12. 2022Flexible text generation for counterfactual fairness probingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Robust Reward Modeling via Causal Rubricscited by 18
  2. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 10
  3. 2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
  4. 2026Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferencescited by 0
  5. 2026HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewardscited by 0
  6. 2026RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
  2. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
  3. 2026Rater State Bias in RLHF Preference Data: An Audit Framework
  4. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  5. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
  6. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  7. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations
  8. 2026Debiasing Reward Models via Causally Motivated Inference-Time Intervention