mlsanityData to 7 Oct 2026

Training Language Models with Language Feedback at Scale

arXiv 2303.16755 ↗First posted 28 Mar 2023cs.CL cs.AI cs.LGcited by 125

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries. Learning from both language and comparison feedback outperforms learning from each alone, achieving human-level summarization performance.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 73 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

49 of 73 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022On Improving Summarization Factual Consistency from Natural Language Feedback20 Dec
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct
  4. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  5. 2022Training Language Models with Language Feedback29 Apr
  6. 2020Learning to summarize from human feedback2 Sep

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
31 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  5. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  6. 2022Self-critiquing models for assisting human evaluatorscited by 372
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  8. 2021True Few-Shot Learning with Language Modelscited by 343
  9. 2022Can language models learn from explanations in context?cited by 248
  10. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  11. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  12. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  13. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  14. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  15. 2022Language Model Cascadescited by 104
  16. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  17. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  18. 2022PEER: A Collaborative Language Modelcited by 86
  19. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  20. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  21. 2022Inferring Rewards from Language in Contextcited by 50
  22. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  23. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  24. 2020Learning Rewards from Linguistic Feedbackcited by 41
  25. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  26. 2021Interactive Learning from Activity Descriptioncited by 26
  27. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  28. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
  29. 2016Dialog-based Language Learningcited by 21
  30. 2018Guide Me: Interacting with Deep Networkscited by 16
  31. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 46; 2024: 45; 2025: 14; 2026: 20.

125 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  9. 2023Large Language Model Alignment: A Surveycited by 241
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  12. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
38 more
  1. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  2. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  3. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  4. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  5. 2024Critique-out-Loud Reward Modelscited by 105
  6. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  8. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  9. 2024Towards Conversational Diagnostic AIcited by 79
  10. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  11. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  12. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  13. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  14. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  15. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  16. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  17. 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
  18. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  19. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  20. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  21. 2023Learning to Generate Better Than Your LLMcited by 49
  22. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  23. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  24. 2024Improving Reward Models with Synthetic Critiquescited by 44
  25. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  26. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  27. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  28. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  29. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  30. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  31. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  32. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  33. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  34. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
  35. 2023SELF: Self-Evolution with Language Feedbackcited by 24
  36. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  37. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  38. 2025Entropy-Gated Branching for Efficient Test-Time Reasoningcited by 21

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Language Models with Demonstrated Feedback
  2. 2024Policy Improvement using Language Feedback Models
  3. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  4. 2023Interpreting Learned Feedback Patterns in Large Language Models
  5. 2023Pretraining Language Models with Human Preferences
  6. 2023Improving Code Generation by Training with Natural Language Feedback
  7. 2023Chain of Hindsight Aligns Language Models with Feedback
  8. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewards