# Leveraging Implicit Feedback from Deployment Data in Dialogue

arXiv: https://arxiv.org/abs/2307.14117 · HTML: https://mlsanity.com/paper/2307.14117/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 95 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 30 of 33)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [TweetNLP: Cutting-Edge Natural Language Processing for Social Media](https://mlsanity.com/paper/2206.14774/) (2022)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [The CRINGE Loss: Learning what language not to model](https://mlsanity.com/paper/2211.05826/) (2022)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize](https://mlsanity.com/paper/1812.10757/) (2018)
- [Assessing Political Prudence of Open-domain Chatbots](https://mlsanity.com/paper/2106.06157/) (2021)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)
- [Learning to Decode for Future Success](https://mlsanity.com/paper/1701.06549/) (2017)
- [Topic-based Evaluation for Conversational Bots](https://mlsanity.com/paper/1801.03622/) (2018)
- [Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning](https://mlsanity.com/paper/2110.06273/) (2021)
- [Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls](https://mlsanity.com/paper/2208.03295/) (2022)
- [Improving Open Language Models by Learning from Organic Interactions](https://mlsanity.com/paper/2306.04707/) (2023)
- [Neural Response Ranking for Social Conversation: A Data-Efficient Approach](https://mlsanity.com/paper/1811.00967/) (2018)

## Built on by (most cited first, 14 of 14)

- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal](https://mlsanity.com/paper/2507.23158/) (2025)
- [The Era of Real-World Human Interaction: RL from User Conversations](https://mlsanity.com/paper/2509.25137/) (2025)
- [Retrospective Learning from Interactions](https://mlsanity.com/paper/2410.13852/) (2024)
- [WildReward: Learning Reward Models from In-the-Wild Human Interactions](https://mlsanity.com/paper/2602.08829/) (2026)
- [ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study](https://mlsanity.com/paper/2401.06513/) (2024)
- [Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition](https://mlsanity.com/paper/2505.15922/) (2025)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production](https://mlsanity.com/paper/2603.01973/) (2026)
- [Image Score: Learning and Evaluating Human Preferences for Mercari Search](https://mlsanity.com/paper/2408.11349/) (2024)
- [Dialogue Systems Engineering: A Survey and Future Directions](https://mlsanity.com/paper/2508.02279/) (2025)
- [Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks](https://mlsanity.com/paper/2605.26440/) (2026)
- [LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks](https://mlsanity.com/paper/2605.27375/) (2026)

## Abstract (arXiv)

We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.