# Learning to Refine with Fine-Grained Natural Language Feedback

arXiv: https://arxiv.org/abs/2407.02397 · HTML: https://mlsanity.com/paper/2407.02397/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 75 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 46)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://mlsanity.com/paper/2305.08844/) (2023)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Shepherd: A Critic for Language Model Generation](https://mlsanity.com/paper/2308.04592/) (2023)

## Built on by (most cited first, 27 of 27)

- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning](https://mlsanity.com/paper/2409.12147/) (2024)
- [Thinking Machines: A Survey of LLM based Reasoning Strategies](https://mlsanity.com/paper/2503.10814/) (2025)
- [HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks](https://mlsanity.com/paper/2503.04378/) (2025)
- [RefineBench: Evaluating Refinement Capability of Language Models via Checklists](https://mlsanity.com/paper/2511.22173/) (2025)
- [Localizing Factual Inconsistencies in Attributable Text Generation](https://mlsanity.com/paper/2410.07473/) (2024)
- [Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing](https://mlsanity.com/paper/2502.18448/) (2025)
- [Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinement](https://mlsanity.com/paper/2603.08251/) (2026)
- [EvalAgent: Discovering Implicit Evaluation Criteria from the Web](https://mlsanity.com/paper/2504.15219/) (2025)
- [MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration](https://mlsanity.com/paper/2503.15272/) (2025)
- [ProRefine: Inference-Time Prompt Refinement with Textual Feedback](https://mlsanity.com/paper/2506.05305/) (2025)
- [Training Language Model to Critique for Better Refinement](https://mlsanity.com/paper/2506.22157/) (2025)
- [ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback](https://mlsanity.com/paper/2503.21332/) (2025)
- [RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models](https://mlsanity.com/paper/2504.11381/) (2025)
- [Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision](https://mlsanity.com/paper/2601.13217/) (2026)
- [The Superalignment of Superhuman Intelligence with Large Language Models](https://mlsanity.com/paper/2412.11145/) (2024)
- [Adaptive Multi-Agent Response Refinement in Conversational Systems](https://mlsanity.com/paper/2511.08319/) (2025)
- [What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioning](https://mlsanity.com/paper/2409.17172/) (2024)
- [LLM-Driven Self-Refinement for Embodied Drone Task Planning](https://mlsanity.com/paper/2508.15501/) (2025)
- [PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise](https://mlsanity.com/paper/2511.01359/) (2025)
- [Learning from Self Critique and Refinement for Faithful LLM Summarization](https://mlsanity.com/paper/2512.05387/) (2025)
- [LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization](https://mlsanity.com/paper/2604.25130/) (2026)
- [Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation](https://mlsanity.com/paper/2608.30397/) (2026)
- [Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding](https://mlsanity.com/paper/2504.01132/) (2025)
- [Detect, Remask, Repair: Diffusion Editing for Faithful Summarization of Evolving Contexts](https://mlsanity.com/paper/2606.12807/) (2026)
- [Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning](https://mlsanity.com/paper/2607.02490/) (2026)
- [Training LLM Judges from Language Feedback via Position-Selective Self-Distillation](https://mlsanity.com/paper/2609.38792/) (2026)

## Abstract (arXiv)

Recent work has explored the capability of large language models (LLMs) to identify and correct errors in LLM-generated responses. These refinement approaches frequently evaluate what sizes of models are able to do refinement for what problems, but less attention is paid to what effective feedback for refinement looks like. In this work, we propose looking at refinement with feedback as a composition of three distinct LLM competencies: (1) detection of bad generations; (2) fine-grained natural language critique generation; (3) refining with fine-grained feedback. The first step can be implemented with a high-performing discriminative model and steps 2 and 3 can be implemented either via prompted or fine-tuned LLMs. A key property of the proposed Detect, Critique, Refine ("DCR") method is that the step 2 critique model can give fine-grained feedback about errors, made possible by offloading the discrimination to a separate model in step 1. We show that models of different capabilities benefit from refining with DCR on the task of improving factual consistency of document grounded summaries. Overall, DCR consistently outperforms existing end-to-end refinement approaches and current trained models not fine-tuned for factuality critiquing.