mlsanityData to 14 Oct 2026

GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements

arXiv 2402.10963 ↗First posted 13 Feb 2024cs.CL cs.LGcited by 103

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

State-of-the-art language models can exhibit impressive reasoning refinement capabilities on math, science or coding tasks. However, recent work demonstrates that even the best models struggle to identify textitwhen and where to refine without access to external feedback. Outcome-based Reward Models (ORMs), trained to predict correctness of the final answer indicating when to refine, offer one convenient solution for deciding when to refine. Process Based Reward Models (PRMs), trained to predict correctness of intermediate steps, can then be used to indicate where to refine. But they are expensive to train, requiring extensive human annotations. In this paper, we propose Stepwise ORMs (SORMs) which are trained, only on synthetic data, to approximate the expected future reward of the optimal policy or Vstar. More specifically, SORMs are trained to predict the correctness of the final answer when sampling the current policy many times (rather than only once as in the case of ORMs). Our experiments show that SORMs can more accurately detect incorrect reasoning steps compared to ORMs, thus improving downstream accuracy when doing refinements. We then train textitglobal refinement models, which take only the question and a draft solution as input and predict a corrected solution, and textitlocal refinement models which also take as input a critique indicating the location of the first reasoning error. We generate training data for both models synthetically by reusing data used to train the SORM. We find combining global and local refinements, using the ORM as a reranker, significantly outperforms either one individually, as well as a best of three sample baseline. With this strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned with RL) on GSM8K from 53% to 65% when greedily sampled.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 40 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

36 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  2. 2023Let's Verify Step by Step31 May
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
20 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  7. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  8. 2022Self-critiquing models for assisting human evaluatorscited by 380
  9. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  10. 2022Generating Sequences by Learning to Self-Correctcited by 277
  11. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  12. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  13. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  14. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  15. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  16. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 112
  17. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  18. 2022PEER: A Collaborative Language Modelcited by 86
  19. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
  20. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 63; 2026: 14.

103 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  4. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  5. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  6. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  7. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  8. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  10. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  11. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  12. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
38 more
  1. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  2. 2025Can Large Reasoning Models Self-Train?cited by 68
  3. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  4. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  5. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 56
  6. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  7. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoningcited by 41
  8. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  9. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  10. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 37
  11. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  12. 2025Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Modelscited by 33
  13. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  14. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 29
  15. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  16. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  17. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  18. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 24
  19. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  20. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 22
  21. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  22. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  23. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
  24. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 17
  25. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  26. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  27. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  28. 2025Empowering GraphRAG with Knowledge Filtering and Integrationcited by 15
  29. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  30. 2024Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviorscited by 13
  31. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  32. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  33. 2025Test-time Prompt Interventioncited by 12
  34. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  35. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 11
  36. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  37. 2025Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agentscited by 10
  38. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?cited by 10

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  2. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2023Let's Reinforce Step by Step
  5. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
  6. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  7. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learning