# RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals

arXiv: https://arxiv.org/abs/2410.11348 · HTML: https://mlsanity.com/paper/2410.11348/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 13 of 13)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior](https://mlsanity.com/paper/2205.14140/) (2022)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens](https://mlsanity.com/paper/2210.14011/) (2022)
- [Zero-shot LLM-guided Counterfactual Generation: A Case Study on NLP Model Evaluation](https://mlsanity.com/paper/2405.04793/) (2024)
- [Causal Estimation for Text Data with (Apparent) Overlap Violations](https://mlsanity.com/paper/2210.00079/) (2022)
- [Flexible text generation for counterfactual fairness probing](https://mlsanity.com/paper/2206.13757/) (2022)

## Built on by (most cited first, 8 of 8)

- [Robust Reward Modeling via Causal Rubrics](https://mlsanity.com/paper/2506.16507/) (2025)
- [CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling](https://mlsanity.com/paper/2603.08035/) (2026)
- [Preference Learning for AI Alignment: a Causal Perspective](https://mlsanity.com/paper/2506.05967/) (2025)
- [Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal](https://mlsanity.com/paper/2606.12360/) (2026)
- [Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences](https://mlsanity.com/paper/2606.07629/) (2026)
- [HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards](https://mlsanity.com/paper/2608.06012/) (2026)
- [RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback](https://mlsanity.com/paper/2609.33989/) (2026)
- [LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games](https://mlsanity.com/paper/2610.04441/) (2026)

## Abstract (arXiv)

Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper we develop Rewrite-based Attribute Treatment Estimator (RATE) as an effective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Importantly, RATE measures the causal effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfactuals examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substantial bias in the estimated sensitivity of the reward model to the attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting twice. We establish the validity of the RATE procedure and show empirically that it is an effective estimator.