mlsanityData to 14 Oct 2026

REFINER: Reasoning Feedback on Intermediate Representations

arXiv 2304.01904 ↗First posted 4 Apr 2023cs.CLcited by 205

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e.g., chain-of-thought prompting. However, these intermediate inference steps may be inappropriate deductions from the initial context and lead to incorrect final predictions. Here we introduce REFINER, a framework for finetuning LMs to explicitly generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. Specifically, the critic provides structured feedback that the reasoning LM uses to iteratively improve its intermediate arguments. Empirical evaluations of REFINER on three diverse reasoning tasks show significant improvements over baseline LMs of comparable scale. Furthermore, when using GPT-3.5 or ChatGPT as the reasoner, the trained critic significantly improves reasoning without finetuning the reasoner. Finally, our critic model is trained without expensive human-in-the-loop data but can be substituted with humans at inference time.

Key facts. Cited by 205 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 49 of its 68 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

49 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  2. 2022Large Language Models Can Self-Improve20 Oct

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
35 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 966
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  4. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  5. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  6. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  7. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  8. 2022Self-critiquing models for assisting human evaluatorscited by 380
  9. 2022Generating Sequences by Learning to Self-Correctcited by 277
  10. 2020Transformers as Soft Reasoners over Languagecited by 270
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  12. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  13. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
  14. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  15. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  16. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  17. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  18. 2020Towards Controllable Biases in Language Generationcited by 112
  19. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  20. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  21. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  22. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51
  23. 2022Training Language Models with Language Feedbackcited by 49
  24. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  25. 2022Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extractioncited by 44
  26. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  27. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  28. 2021NL-EDIT: Correcting semantic parse errors through natural language interactioncited by 33
  29. 2021Interactive Learning from Activity Descriptioncited by 26
  30. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  31. 2016Dialog-based Language Learningcited by 21
  32. 2018Guide Me: Interacting with Deep Networkscited by 16
  33. 2021COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completioncited by 12
  34. 2019Improving Natural Language Interaction with Robots Using Advicecited by 9
  35. 2022Argumentative Reward Learning: Reasoning About Human Preferencescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 47; 2024: 83; 2025: 57; 2026: 18.

205 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  8. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  9. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  10. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  11. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  12. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
38 more
  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  2. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  3. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  4. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  5. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 159
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 124
  9. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  10. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  11. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  12. 2024Large Language Models Meet NLP: A Surveycited by 103
  13. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 102
  14. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  15. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
  16. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  17. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  18. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
  19. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  20. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  21. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  22. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 77
  23. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  24. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  25. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  26. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  27. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  28. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 60
  29. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  30. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  31. 2023Prompt Engineering a Prompt Engineercited by 53
  32. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52
  33. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  34. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 51
  35. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  36. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  37. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  38. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49

50 most cited of 205 All 205 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
  2. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
  3. 2025First Try Matters: Revisiting the Role of Reflection in Reasoning Models
  4. 2025Efficient Post-Training Refinement of Latent Reasoning in Large Language Models
  5. 2026Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost
  6. 2026Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
  7. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models
  8. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate