mlsanityData to 7 Oct 2026

DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity

arXiv 2606.09043 ↗First posted 8 Jun 2026cs.LG cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models trained from pairwise preferences often exploit superficial shortcut cues rather than learning true response quality. We propose DynaCF, a dynamic reweighting framework for mitigating shortcut learning in reward model training. Unlike static shortcut heuristics, DynaCF measures shortcut sensitivity online during optimization by applying semantics-preserving counterfactual perturbations and tracking the resulting margin shifts and preference flips under the current model. Samples with higher shortcut sensitivity are dynamically downweighted in the Bradley-Terry objective, encouraging the model to rely less on superficial patterns and more on task-relevant preference signals. Extensive experiments show that DynaCF consistently improves robustness in preference modeling.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

31 of 39 references.

Closest ancestors · 23

Same direction, newest first

  1. 2026AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling13 Jan
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025Robust Reward Modeling via Causal Rubrics19 Jun
  4. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  5. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  6. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  8. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  9. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  10. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  12. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
11 more
  1. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 8

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  8. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learningcited by 0
  2. 2026Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learning
  2. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning
  3. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
  4. 2025ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic Optimization
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  6. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  7. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  8. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling