mlsanityData to 7 Oct 2026

What do Reward Models Memorize?

arXiv 2607.24484 ↗First posted 27 Jul 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.g., model identity, user sampling strategy), and 3) overgeneralize simple heuristic correlates of human preference (e.g., length, compliance) when confronted with unseen preference pairs. Overall, our findings indicate that discriminative training of RMs from human preference data results in biased RMs not yet capable of judging response quality in context-dependent scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 71 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

42 of 71 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks19 Mar
  2. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learning21 Oct
  3. 2025Reward Models are Metrics in a Trench Coat3 Oct
  4. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  5. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback26 May
  6. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scores14 Apr
  7. 2025Reward Models Identify Consistency, Not Causality20 Feb
  8. 2024Post-hoc Reward Calibration: A Case Study on Length Bias25 Sep
  9. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  11. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
3 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  8. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  9. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  10. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
  11. 2024Rule Based Rewards for Language Model Safetycited by 106
  12. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
15 more
  1. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  2. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  3. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  4. 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
  5. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  6. 2022An Empirical Study of Memorization in NLPcited by 21
  7. 2019MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversationscited by 20
  8. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  9. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 16
  10. 2025The AI Gap: How Socioeconomic Status Affects Language Technology Interactionscited by 15
  11. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  12. 2022The Curious Case of Benign Memorizationcited by 11
  13. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reasoncited by 11
  14. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  15. 2025Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correctioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  2. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  3. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
  4. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  5. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  6. 2025Debiasing Reward Models by Representation Learning with Guarantees
  7. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  8. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysis