mlsanityData to 7 Oct 2026

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

arXiv 1901.05415 ↗First posted 16 Jan 2019cs.CL cs.AI cs.HCcited by 102

Not on the map. Why?

Cited by 32 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in conversation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user's responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot's dialogue abilities further. On the PersonaChat chit-chat dataset with over 131k training examples, we find that learning from dialogue with a self-feeding chatbot significantly improves performance, regardless of the amount of traditional supervision.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 3; 2021: 19; 2022: 18; 2023: 21; 2024: 18; 2025: 15; 2026: 6.

102 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
38 more
  1. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  2. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  3. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  4. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  5. 2023Training Language Models with Language Feedback at Scalecited by 125
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  8. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  9. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  10. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  11. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  12. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  13. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  14. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  15. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  16. 2022Training Language Models with Language Feedbackcited by 48
  17. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  18. 2021Grounding 'Grounding' in NLPcited by 44
  19. 2021Putting Humans in the Natural Language Processing Loop: A Surveycited by 40
  20. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  21. 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
  22. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 35
  23. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  24. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  25. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
  26. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  27. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20
  28. 2023Learning to Simulate Natural Language Feedback for Interactive Semantic Parsingcited by 19
  29. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processingcited by 19
  30. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  31. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18
  32. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 17
  33. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  34. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  35. 2024Naturally Occurring Feedback is Common, Extractable and Usefulcited by 14
  36. 2024Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversationscited by 13
  37. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  38. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12

50 most cited of 102 All 102 (JSON)