mlsanityData to 14 Oct 2026

Language Models that Think, Chat Better

arXiv 2509.20357 ↗First posted 24 Sep 2025cs.CLcited by 24

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) trains language models to use long chain-of-thought reasoning (CoT) in domains like mathematics and code with rule-based verifiers. However, long CoT learned through RLVR does not generalize well to open-ended tasks -- such as writing essay outlines or making meal plans -- where humans reason routinely. This paper establishes the benefits of long CoT for general-purpose chat capabilities and introduces RL with Model-rewarded Thinking (RLMT)1, which pushes RLVR beyond verifiable domains. Using diverse real-world prompts, RLMT requires LMs to generate long CoT reasoning before responding, and optimizes them with online RL against a preference-based reward model used in RLHF. Across 40 training runs on Llama-3.1-8B and Qwen-2.5-7B (both base and instruct) and multiple optimization algorithms (DPO, PPO, and GRPO), RLMT consistently outperforms standard RLHF pipelines. This includes substantial gains of 3-7 points on three chat benchmarks (AlpacaEval2, WildBench, and ArenaHardV2), along with 1-3 point improvements on other tasks like creative writing and general knowledge. RLMT can also be applied directly to base models without an SFT stage, akin to DeepSeek-R1-Zero. Remarkably, with only 7K prompts, Llama-3.1-8B base trained with our RLMT recipe outperforms Llama-3.1-8B-Instruct post-trained with a complex multi-staged pipeline with 25M+ examples. We close with qualitative and quantitative analyses of how trained models plan their responses. Our results rethink the post-training pipeline and call upon future work to understand and employ thinking more broadly.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 79 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

17 of 79 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  2. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning21 May
  3. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb

Other references · 10

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  5. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  6. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  7. 2025BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillationcited by 23
  8. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  9. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  10. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 18.

24 papers.

Sort:

By area:

  1. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  2. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  3. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  4. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  5. 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 9
  6. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  7. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  8. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 7
  9. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  10. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  11. 2025Zero Reinforcement Learning Towards General Domainscited by 5
  12. 2026Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learningcited by 5
12 more
  1. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
  2. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  3. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  4. 2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 3
  5. 2026Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learningcited by 1
  6. 2026MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Modelscited by 1
  7. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 1
  8. 2026Do Reasoning Models Enhance Embedding Models?cited by 0
  9. 2026SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructionscited by 0
  10. 2026CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflictscited by 0
  11. 2026Don't Wait to Reply: Towards Responsive yet Thoughtful Dialogue through Proactive Thinkingcited by 0
  12. 2026Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
  3. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs
  4. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learning
  5. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  6. 2026RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
  7. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  8. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example