mlsanityData to 14 Oct 2026

CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria

arXiv 2601.20327 ↗First posted 28 Jan 2026cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Automatic evaluation is crucial yet challenging for open-ended natural language generation, especially when rule-based metrics are infeasible. Compared with traditional methods, the recent LLM-as-a-Judge paradigms enable better and more flexible evaluation, and show promise as generative reward models for reinforcement learning. However, prior work has revealed a notable gap between their seemingly impressive benchmark performance and actual effectiveness in RL practice. We attribute this issue to some limitations in existing studies, including the dominance of pairwise evaluation and inadequate optimization of evaluation criteria. Therefore, we propose CE-RM-4B, a pointwise generative reward model trained with a dedicated two-stage rollout method, and adopting unified query-based criteria. Using only about 5.7K high-quality data curated from the open-source preference dataset, our CE-RM-4B achieves superior performance on diverse reward model benchmarks, especially in Best-of-N scenarios, and delivers more effective improvements in downstream RL practice.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 45 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

26 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  2. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge20 May
  3. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning5 May
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  7. 2024How to Evaluate Reward Models for RLHF18 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  11. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  12. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
6 more
  1. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  2. 2025Reward Reasoning Modelcited by 47
  3. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  4. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  5. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  6. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  2. 2026Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  2. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generation
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
  5. 2026RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
  6. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  7. 2025Towards Cost-Effective Reward Guided Text Generation
  8. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation