mlsanityData to 7 Oct 2026

Training Language Models with Language Feedback

arXiv 2204.14146 ↗First posted 29 Apr 2022cs.CL cs.AI cs.LGcited by 48

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: comparisons between pairs of model-generated task outputs. Comparison feedback conveys limited information about human preferences per human evaluation. Here, we propose to learn from natural language feedback, which conveys more information per human evaluation. We learn from language feedback on model outputs using a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements, finding that only large language models (175B parameters) do so. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization ability.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 43 references that are in the map. First posted on arXiv 29 Apr 2022.

Builds on

37 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
24 more
  1. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  4. 2022Can language models learn from explanations in context?cited by 248
  5. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  6. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  7. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  8. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  9. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  10. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  11. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  12. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  13. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  14. 2022Inferring Rewards from Language in Contextcited by 50
  15. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  16. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  17. 2020Learning Rewards from Linguistic Feedbackcited by 41
  18. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  19. 2021Interactive Learning from Activity Descriptioncited by 26
  20. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  21. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
  22. 2016Dialog-based Language Learningcited by 21
  23. 2018Guide Me: Interacting with Deep Networkscited by 16
  24. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 24; 2024: 8; 2025: 4; 2026: 3.

48 papers.

Sort:

By area:

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  4. 2022Generating Sequences by Learning to Self-Correctcited by 271
  5. 2023Pretraining Language Models with Human Preferencescited by 245
  6. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  7. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  8. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  9. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  10. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  11. 2023Training Language Models with Language Feedback at Scalecited by 125
  12. 2022Language Model Cascadescited by 104
36 more
  1. 2022Evaluating Human-Language Model Interactioncited by 90
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  3. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  4. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  5. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  6. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  7. 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 39
  8. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  9. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  10. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  11. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  12. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22
  13. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20
  14. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  15. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  16. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  17. 2024Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistantscited by 10
  18. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  19. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
  20. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Modelscited by 9
  21. 2023SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcriptscited by 8
  22. 2023Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Contentcited by 7
  23. 2023The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Modelscited by 7
  24. 2025Aligning Instruction Tuning with Pre-trainingcited by 7
  25. 2023CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistantscited by 5
  26. 2023Solving NLP Problems through Human-System Collaboration: A Discussion-based Approachcited by 5
  27. 2024Towards Aligning Language Models with Textual Feedbackcited by 5
  28. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  29. 2022Manifestations of Xenophobia in AI Systemscited by 4
  30. 2023System-Level Natural Language Feedbackcited by 4
  31. 2023Extrinsically-Focused Evaluation of Omissions in Medical Summarizationcited by 4
  32. 2023Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learningcited by 2
  33. 2024Rules, Cases, and Reasoning: Positivist Legal Theory as a Framework for Pluralistic AI Alignmentcited by 2
  34. 2026SIEVE: Sample-Efficient Parametric Learning from Natural Languagecited by 2
  35. 2024Aligning language models with human preferencescited by 1
  36. 2026Efficiently Aligning Language Models with Online Natural Language Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Human Feedback is not Gold Standard
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  3. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  4. 2022Self-critiquing models for assisting human evaluators
  5. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  6. 2023Self-Refine: Iterative Refinement with Self-Feedback
  7. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  8. 2025Maximizing Signal in Human-Model Preference Alignment