mlsanityData to 7 Oct 2026

Robust Reward Modeling for Large Language Models via Causal Decomposition

arXiv 2604.13833 ↗First posted 15 Apr 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 32 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

20 of 32 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Robust Reward Modeling via Causal Rubrics19 Jun
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  7. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  8. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  9. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  8. 2013k-Sparse Autoencoderscited by 209
  9. 2020A Survey on Causal Inferencecited by 202
  10. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
  2. 2023Reward Collapse in Aligning Large Language Models
  3. 2026Automatically Finding Reward Model Biases
  4. 2026Small Reward Models via Backward Inference
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  6. 2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generation
  7. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  8. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment