mlsanityData to 7 Oct 2026

Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models

arXiv 2603.20212 ↗First posted 2 Mar 2026cs.CL cs.LGcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: scalar-style first-token pairwise judgment (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. Under hybrid inference, F/S-RM achieves state-of-the-art accuracy with an average score of 84.3 across benchmarks, while reducing token consumption by 22.5%.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 32 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

21 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Think Twice: Branch-and-Rethink Reasoning Reward Model27 Oct
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025Reward Reasoning Model20 May
  4. 2025RM-R1: Reward Modeling as Reasoning5 May
  5. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug

Other references · 14

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  5. 2025Deep Think with Confidencecited by 199
  6. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  8. 2024Generative Reward Modelscited by 148
  9. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  10. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  11. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  12. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
2 more
  1. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  2. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
  2. 2026Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
  3. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
  4. 2025Rethinking Reward Models for Multi-Domain Test-Time Scaling
  5. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
  6. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  8. 2025Libra: Assessing and Improving Reward Model by Learning to Think