mlsanityData to 7 Oct 2026

RLVF: Learning from Verbal Feedback without Overgeneralization

arXiv 2402.10893 ↗First posted 16 Feb 2024cs.LG cs.AI cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a model with such feedback leads to overgeneralization of the feedback to contexts where it is not relevant. We study the problem of incorporating verbal feedback without such overgeneralization, inspiring a new method Contextualized Critiques with Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level feedback to generate a small synthetic preference dataset specifying how the feedback should (and should not) be applied. It then fine-tunes the model in accordance with the synthetic preference data while minimizing the divergence from the original model for prompts where the feedback does not apply. Our experimental results indicate that our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts. For both human- and GPT-4-generated high-level feedback, C3PO effectively adheres to the given feedback comparably to in-context baselines while reducing overgeneralization by 30%.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 44 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

26 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2021Fast Model Editing at Scalecited by 491
  11. 2022Memory-Based Model Editing at Scalecited by 443
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
12 more
  1. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  2. 2020Transformers as Soft Reasoners over Languagecited by 265
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  4. 2020Editable Neural Networkscited by 154
  5. 2022Learning by Distilling Contextcited by 128
  6. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  7. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  8. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  9. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  10. 2022Fixing Model Bugs with Natural Language Patchescited by 32
  11. 2023Editing Personality for Large Language Modelscited by 22
  12. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 7.

18 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  3. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  4. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  5. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  6. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  7. 2026Aligning Language Models from User Interactionscited by 21
  8. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  9. 2025Characterizing Photorealism and Artifacts in Diffusion Model-Generated Imagescited by 13
  10. 2026HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillationcited by 8
  11. 2025Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimizationcited by 7
  12. 2025Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiencescited by 7
6 more
  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  2. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  3. 2025Exploring the Secondary Risks of Large Language Modelscited by 1
  4. 2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1
  5. 2026Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learningcited by 1
  6. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  2. 2025Reinforcement Learning from User Feedback
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  4. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  5. 2024Generative Reward Models
  6. 2023Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2023Training Language Models with Language Feedback at Scale