mlsanityData to 7 Oct 2026

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

arXiv 2210.15893 ↗First posted 28 Oct 2022cs.CL cs.AIcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 29 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

24 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  2. 2022Training Language Models with Language Feedback29 Apr

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2020The Pushshift Reddit Datasetcited by 289
  9. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  10. 2021Internet-Augmented Dialogue Generationcited by 217
  11. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  12. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
10 more
  1. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  2. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  3. 2022PEER: A Collaborative Language Modelcited by 86
  4. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  5. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  6. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  7. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  8. 2021NL-EDIT: Correcting semantic parse errors through natural language interactioncited by 33
  9. 2020Deploying Lifelong Open-Domain Dialogue Learningcited by 17
  10. 2022Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trollscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 4; 2025: 1; 2026: 1.

20 papers.

Sort:

By area:

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  4. 2023Training Language Models with Language Feedback at Scalecited by 125
  5. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  6. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  7. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  8. 2023Specific versus General Principles for Constitutional AIcited by 41
  9. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  10. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  11. 2023DUnE: Dataset for Unified Editingcited by 19
  12. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
8 more
  1. 2024Naturally Occurring Feedback is Common, Extractable and Usefulcited by 14
  2. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  3. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Modelscited by 9
  4. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9
  5. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  6. 2023System-Level Natural Language Feedbackcited by 4
  7. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perceptioncited by 2
  8. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?
  2. 2022Improving alignment of dialogue agents via targeted human judgements
  3. 2022Towards Boosting the Open-Domain Chatbot with Human Feedback
  4. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  5. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data
  6. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
  7. 2023SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback
  8. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents