# Dialogue Learning With Human-In-The-Loop

arXiv: https://arxiv.org/abs/1611.09823 · HTML: https://mlsanity.com/paper/1611.09823/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2016.

## Built on by (most cited first, 30 of 46)

- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [A data-driven approach for learning to control computers](https://mlsanity.com/paper/2202.08137/) (2022)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [Learning Rewards from Linguistic Feedback](https://mlsanity.com/paper/2009.14715/) (2020)
- [Putting Humans in the Natural Language Processing Loop: A Survey](https://mlsanity.com/paper/2103.04044/) (2021)
- [Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback](https://mlsanity.com/paper/2208.03270/) (2022)
- [Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment](https://mlsanity.com/paper/2204.03025/) (2022)
- [LeTI: Learning to Generate from Textual Interactions](https://mlsanity.com/paper/2305.10314/) (2023)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances](https://mlsanity.com/paper/2502.16804/) (2025)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)
- [DUnE: Dataset for Unified Editing](https://mlsanity.com/paper/2311.16087/) (2023)
- [RLVF: Learning from Verbal Feedback without Overgeneralization](https://mlsanity.com/paper/2402.10893/) (2024)
- [Reasons to Reject? Aligning Language Models with Judgments](https://mlsanity.com/paper/2312.14591/) (2023)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)

## Abstract (arXiv)

An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.