mlsanityData to 7 Oct 2026

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

arXiv 2305.08844 ↗First posted 15 May 2023cs.CLcited by 104

Not on the map. Why?

Cited by 31 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

32 of 50 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2021Fast Model Editing at Scalecited by 491
  11. 2022Memory-Based Model Editing at Scalecited by 443
  12. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
20 more
  1. 2022Self-critiquing models for assisting human evaluatorscited by 372
  2. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  3. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  4. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2022Language Model Cascadescited by 104
  7. 2022PEER: A Collaborative Language Modelcited by 86
  8. 2020Felix: Flexible Text Editing Through Tagging and Insertioncited by 58
  9. 2019Countering Language Drift via Visual Groundingcited by 55
  10. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  11. 2019Corpora Generation for Grammatical Error Correctioncited by 53
  12. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  13. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  14. 2020Text Editing by Commandcited by 33
  15. 2022Fixing Model Bugs with Natural Language Patchescited by 32
  16. 2022Learning to Model Editing Processescited by 30
  17. 2022Text Generation with Text-Editing Modelscited by 21
  18. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20
  19. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  20. 2021Interscript: A dataset for interactive learning of scripts through error feedbackcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 40; 2025: 33; 2026: 5.

104 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  3. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  10. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  11. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  12. 2024Critique-out-Loud Reward Modelscited by 105
38 more
  1. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  3. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  4. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  5. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  6. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  7. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  8. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 49
  9. 2023Learning from models beyond fine-tuningcited by 47
  10. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  11. 2024RaFe: Ranking Feedback Improves Query Rewriting for RAGcited by 47
  12. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  13. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  14. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  15. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  16. 2024Direct Judgement Preference Optimizationcited by 33
  17. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 33
  18. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  19. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30
  20. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  21. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  22. 2025LLMs Can Generate a Better Answer by Aggregating Their Own Responsescited by 27
  23. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  24. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  25. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  26. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  27. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  28. 2023Small Language Models Improve Giants by Rewriting Their Outputscited by 18
  29. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  30. 2024LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robotscited by 18
  31. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  32. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  33. 2024Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planningcited by 16
  34. 2024Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinementcited by 16
  35. 2024Prompt Chaining or Stepwise Prompt? Refinement in Text Summarizationcited by 16
  36. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  37. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  38. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  4. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
  5. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  6. 2023Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
  7. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  8. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language