mlsanityData to 7 Oct 2026

LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling

arXiv 2510.06915 ↗First posted 8 Oct 2025cs.CL cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward model (RM) plays a pivotal role in aligning large language model (LLM) with human preferences. As real-world applications increasingly involve long history trajectories, e.g., LLM agent, it becomes indispensable to evaluate whether a model's responses are not only high-quality but also grounded in and consistent with the provided context. Yet, current RMs remain confined to short-context settings and primarily focus on response-level attributes (e.g., safety or helpfulness), while largely neglecting the critical dimension of long context-response consistency. In this work, we introduce Long-RewardBench, a benchmark specifically designed for long-context RM evaluation, featuring both Pairwise Comparison and Best-of-N tasks. Our preliminary study reveals that even state-of-the-art generative RMs exhibit significant fragility in long-context scenarios, failing to maintain context-aware preference judgments. Motivated by the analysis of failure patterns observed in model outputs, we propose a general multi-stage training strategy that effectively scales arbitrary models into robust Long-context RMs (LongRMs). Experiments show that our approach not only substantially improves performance on long-context evaluation but also preserves strong short-context capability. Notably, our 8B LongRM outperforms much larger 70B-scale baselines and matches the performance of the proprietary Gemini 2.5 Pro model.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 58 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

44 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Pre-Trained Policy Discriminators are General Reward Models7 Jul
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025Reward Models in Deep Reinforcement Learning: A Survey18 Jun
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr
  5. 2024LongReward: Improving Long-context Large Language Models with AI Feedback28 Oct
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  8. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Data1 Oct
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  12. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  12. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
18 more
  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  2. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  4. 2023Generative Judge for Evaluating Alignmentcited by 163
  5. 2024Free Process Rewards without Process Labelscited by 162
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  7. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  8. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  9. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
  10. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  11. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  12. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  13. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  14. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedbackcited by 18
  15. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  16. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
  17. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  18. 2023Self-supervised network distillation: an effective approach to exploration in sparse reward environmentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 7
  2. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  4. 2026Reward Modeling for Scientific Writing Evaluationcited by 2
  5. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Context-Aware Preference Modeling for Language Models
  2. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
  3. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  4. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  5. 2024Interpreting Language Reward Models via Contrastive Explanations
  6. 2026Reward Models Inherit Value Biases from Pretraining
  7. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  8. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences