mlsanityData to 14 Oct 2026

Real-Time Aligned Reward Model beyond Semantics

arXiv 2601.22664 ↗First posted 30 Jan 2026cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 89 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

64 of 89 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback28 Mar
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  4. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  6. 2024Fine-Tuning Language Models with Reward Learning on Policy28 Mar
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  9. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  10. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  11. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
6 more
  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  3. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2025Process Reinforcement through Implicit Rewardscited by 504
  12. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
34 more
  1. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  2. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  3. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  4. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  5. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  6. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 159
  7. 2024Nemotron-4 340B Technical Reportcited by 144
  8. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  9. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  10. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  11. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  12. 2021EE-Net: Exploitation-Exploration Neural Networks in Contextual Banditscited by 44
  13. 2025ARM: Adaptive Reasoning Modelcited by 39
  14. 2022SparseTT: Visual Tracking with Sparse Transformerscited by 36
  15. 2026Your Group-Relative Advantage Is Biasedcited by 34
  16. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  17. 2024LLM-Forest: Ensemble Learning of LLMs with Graph-Augmented Prompts for Data Imputationcited by 25
  18. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 24
  19. 2025Pathology-Aware Prototype Evolution via LLM-Driven Semantic Disambiguation for Multicenter Diabetic Retinopathy Diagnosiscited by 22
  20. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  21. 2025Interpretable Unsupervised Joint Denoising and Enhancement for Real-World low-light Scenarioscited by 11
  22. 2024Neural Active Learning Beyond Banditscited by 10
  23. 2025Hyperbolic Diffusion Recommender Modelcited by 9
  24. 2025LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Samplingcited by 9
  25. 2025SpecExit: Accelerating Large Reasoning Model via Speculative Exitcited by 9
  26. 2024FairDgcl: Fairness-aware Recommendation with Dynamic Graph Contrastive Learningcited by 8
  27. 2025LLMBoost: Make Large Language Models Stronger with Boostingcited by 8
  28. 2026PRPO: Aligning Process Reward with Outcome Reward in Policy Optimizationcited by 7
  29. 2023Predicting Text Preference Via Structured Comparative Reasoningcited by 6
  30. 2025Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillationcited by 6
  31. 2025SeeDNorm: Self-Rescaled Dynamic Normalizationcited by 6
  32. 2025Prompt-SID: Learning Structural Representation Prompt via Latent Diffusion for Single-Image Denoisingcited by 5
  33. 2025Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Servicescited by 3
  34. 2025H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneitycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 15
  2. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  3. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  4. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  5. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
  6. 2026Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modalitycited by 1
  7. 2026Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modelingcited by 0
  8. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  9. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvementcited by 0
  10. 2026EchoStyle: Unlocking High-Fidelity Video Stylization with Reverse Data Synthesiscited by 0
  11. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  12. 2026Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Trackingcited by 0
1 more
  1. 2026SquidAgent: Parallelize Wisely, Coordinate Efficientlycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Model Overoptimisation in Iterated RLHF
  2. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  5. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  6. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  7. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs