mlsanityData to 7 Oct 2026

Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning

arXiv 1812.03509 ↗First posted 9 Dec 2018cs.CL cs.AI cs.LGcited by 21

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

The performance of adversarial dialogue generation models relies on the quality of the reward signal produced by the discriminator. The reward signal from a poor discriminator can be very sparse and unstable, which may lead the generator to fall into a local optimum or to produce nonsense replies. To alleviate the first problem, we first extend a recently proposed adversarial dialogue generation method to an adversarial imitation learning solution. Then, in the framework of adversarial inverse reinforcement learning, we propose a new reward model for dialogue generation that can provide a more accurate and precise reward signal for generator training. We evaluate the performance of the resulting model with automatic metrics and human evaluations in two annotation settings. Our experimental results demonstrate that our model can generate more high-quality responses and achieve higher overall performance than the state-of-the-art.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 32 references that are in the map. First posted on arXiv 9 Dec 2018.

Builds on

15 of 32 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  8. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  9. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  10. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  11. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
3 more
  1. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  2. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  3. 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98

Built on by

Cited by year

Citing papers per year: 2019: 1; 2020: 3; 2021: 7; 2022: 3; 2023: 2; 2024: 2; 2025: 3.

21 papers.

Sort:

By area:

  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  3. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  4. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  5. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  6. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  7. 2020Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Explorationcited by 13
  8. 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
  9. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  10. 2021Modulating Language Models with Emotionscited by 9
  11. 2021DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialoguescited by 7
  12. 2022New Challenges in Reinforcement Learning: A Survey of Security and Privacycited by 7
9 more
  1. 2020Online Observer-Based Inverse Reinforcement Learningcited by 4
  2. 2021Improving Response Quality with Backward Reasoning in Open-domain Dialogue Systemscited by 4
  3. 2021DYPLODOC: Dynamic Plots for Document Classificationcited by 4
  4. 2021An Adversarial Learning based Multi-Step Spoken Language Understanding System through Human-Computer Interactioncited by 3
  5. 2021Actionable Entities Recognition Benchmark for Interactive Fictioncited by 2
  6. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1
  7. 2025Limitation Learning: Catching Adverse Dialog with GAILcited by 0
  8. 2025Memory-Driven Self-Improvement for Decision Making with Large Language Modelscited by 0
  9. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Guided Dialog Policy Learning without Adversarial Learning in the Loop
  2. 2021Integrating Pretrained Language Model for Dialogue Policy Learning
  3. 2020Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
  4. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog
  5. 2026Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models
  6. 2021Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management
  7. 2018Toward Diverse Text Generation with Inverse Reinforcement Learning
  8. 2020Inverse Reinforcement Learning with Natural Language Goals