# LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

arXiv: https://arxiv.org/abs/2311.09336 · HTML: https://mlsanity.com/paper/2311.09336/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis](https://mlsanity.com/paper/2210.05035/) (2022)
- [SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes](https://mlsanity.com/paper/2212.09305/) (2022)

## Built on by (most cited first, 30 of 39)

- [VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data](https://mlsanity.com/paper/2502.06737/) (2025)
- [DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines](https://mlsanity.com/paper/2312.13382/) (2023)
- [Learning to Refine with Fine-Grained Natural Language Feedback](https://mlsanity.com/paper/2407.02397/) (2024)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations](https://mlsanity.com/paper/2404.07851/) (2024)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [xTower: A Multilingual LLM for Explaining and Correcting Translation Errors](https://mlsanity.com/paper/2406.19482/) (2024)
- [Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translation](https://mlsanity.com/paper/2503.04554/) (2025)
- [No Need for Explanations: LLMs can implicitly learn from mistakes in-context](https://mlsanity.com/paper/2502.08550/) (2025)
- [MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration](https://mlsanity.com/paper/2503.15272/) (2025)
- [EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning](https://mlsanity.com/paper/2603.21728/) (2026)
- [Visual Prompting with Iterative Refinement for Design Critique Generation](https://mlsanity.com/paper/2412.16829/) (2024)
- [Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement](https://mlsanity.com/paper/2504.05614/) (2025)
- [Localizing and Mitigating Errors in Long-form Question Answering](https://mlsanity.com/paper/2407.11930/) (2024)
- [Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data](https://mlsanity.com/paper/2410.11056/) (2024)
- [Automatic Input Rewriting Improves Translation with Large Language Models](https://mlsanity.com/paper/2502.16682/) (2025)
- [Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation](https://mlsanity.com/paper/2502.19941/) (2025)
- [ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback](https://mlsanity.com/paper/2503.21332/) (2025)
- [EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention](https://mlsanity.com/paper/2508.16771/) (2025)
- [Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue](https://mlsanity.com/paper/2508.05283/) (2025)
- [Uncovering Factor Level Preferences to Improve Human-Model Alignment](https://mlsanity.com/paper/2410.06965/) (2024)
- [From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set](https://mlsanity.com/paper/2411.15387/) (2024)
- [SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation](https://mlsanity.com/paper/2506.00319/) (2025)
- [Reranking-based Generation for Unbiased Perspective Summarization](https://mlsanity.com/paper/2506.15925/) (2025)
- [Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agents](https://mlsanity.com/paper/2510.05188/) (2025)
- [Adaptive Multi-Agent Response Refinement in Conversational Systems](https://mlsanity.com/paper/2511.08319/) (2025)
- [What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation](https://mlsanity.com/paper/2605.13368/) (2026)
- [In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement](https://mlsanity.com/paper/2410.03124/) (2024)
- [Improving Next Tokens via Second-to-Last Predictions with Generate and Refine](https://mlsanity.com/paper/2411.15661/) (2024)

## Abstract (arXiv)

Recent large language models (LLM) are leveraging human feedback to improve their generation quality. However, human feedback is costly to obtain, especially during inference. In this work, we propose LLMRefine, an inference time optimization method to refine LLM's output. The core idea is to use a learned fine-grained feedback model to pinpoint defects and guide LLM to refine them iteratively. Using original LLM as a proposal of edits, LLMRefine searches for defect-less text via simulated annealing, trading off the exploration and exploitation. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.