# Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

arXiv: https://arxiv.org/abs/1609.00777 · HTML: https://mlsanity.com/paper/1609.00777/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2016.

## Built on by (most cited first, 30 of 36)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [Conversational Question Answering: A Survey](https://mlsanity.com/paper/2106.00874/) (2021)
- [LLMRec: Benchmarking Large Language Models on Recommendation Task](https://mlsanity.com/paper/2308.12241/) (2023)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games](https://mlsanity.com/paper/2310.01468/) (2023)
- [ChatShop: Interactive Information Seeking with Language Agents](https://mlsanity.com/paper/2404.09911/) (2024)
- [Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation](https://mlsanity.com/paper/2401.07382/) (2024)
- [Process-Supervised Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2502.01715/) (2025)
- [Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning](https://mlsanity.com/paper/2208.13663/) (2022)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1811.07550/) (2018)
- [Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers](https://mlsanity.com/paper/2103.16289/) (2021)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [Conversational Agents: Theory and Applications](https://mlsanity.com/paper/2202.03164/) (2022)
- [Conversational Question Answering with Reformulations over Knowledge Graph](https://mlsanity.com/paper/2312.17269/) (2023)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- [Locally Aggregated Feature Attribution on Natural Language Model Understanding](https://mlsanity.com/paper/2204.10893/) (2022)
- [REKnow: Enhanced Knowledge for Joint Entity and Relation Extraction](https://mlsanity.com/paper/2206.05123/) (2022)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [NLU for Game-based Learning in Real: Initial Evaluations](https://mlsanity.com/paper/2205.13754/) (2022)
- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [Successor Features for Efficient Multisubject Controlled Text Generation](https://mlsanity.com/paper/2311.04921/) (2023)
- [Converse, Focus and Guess -- Towards Multi-Document Driven Dialogue](https://mlsanity.com/paper/2102.02435/) (2021)
- [Training like Playing: A Reinforcement Learning And Knowledge Graph-based framework for building Automatic Consultation System in Medical Field](https://mlsanity.com/paper/2106.07502/) (2021)
- [A Neural Conversation Generation Model via Equivalent Shared Memory Investigation](https://mlsanity.com/paper/2108.09164/) (2021)
- [Feudal Reinforcement Learning by Reading Manuals](https://mlsanity.com/paper/2110.06477/) (2021)
- [Knowledge Graph-enhanced Sampling for Conversational Recommender System](https://mlsanity.com/paper/2110.06637/) (2021)

## Abstract (arXiv)

This paper proposes KB-InfoBot -- a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve entries based on their attributes. However, such symbolic operations break the differentiability of the system and prevent end-to-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards personalized dialogue agents. The source code is available at https://github.com/MiuLab/KB-InfoBot.