mlsanityData to 14 Oct 2026

Leveraging Implicit Feedback from Deployment Data in Dialogue

arXiv 2307.14117 ↗First posted 26 Jul 2023cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 33 of its 95 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

33 of 95 references.

References · 33

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  9. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  10. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  11. 2018Neural Approaches to Conversational AIcited by 165
  12. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
21 more
  1. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  2. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  3. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  4. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  5. 2022TweetNLP: Cutting-Edge Natural Language Processing for Social Mediacited by 71
  6. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  7. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  8. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  9. 2022Reward Gaming in Conditional Text Generationcited by 30
  10. 2018Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prizecited by 29
  11. 2021Assessing Political Prudence of Open-domain Chatbotscited by 20
  12. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20
  13. 2017Learning to Decode for Future Successcited by 16
  14. 2018Topic-based Evaluation for Conversational Botscited by 16
  15. 2021Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learningcited by 11
  16. 2022Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trollscited by 11
  17. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  18. 2018Neural Response Ranking for Social Conversation: A Data-Efficient Approachcited by 9
  19. 2018Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partnercited by 7
  20. 2021Amortized Noisy Channel Neural Machine Translationcited by 4
  21. 2019Multi-Turn Beam Search for Neural Dialogue Modelingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 4.

14 papers.

Sort:

By area:

  1. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  2. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  3. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  4. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  5. 2024Retrospective Learning from Interactionscited by 6
  6. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  7. 2024ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Studycited by 2
  8. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decompositioncited by 2
  9. 2024Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedbackcited by 1
  10. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  11. 2024Image Score: Learning and Evaluating Human Preferences for Mercari Searchcited by 0
  12. 2025Dialogue Systems Engineering: A Survey and Future Directionscited by 0
2 more
  1. 2026Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Taskscited by 0
  2. 2026LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Taskscited by 0