# Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.07873 · HTML: https://mlsanity.com/paper/2209.07873/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Can Neural Machine Translation be Improved with User Feedback?](https://mlsanity.com/paper/1804.05958/) (2018)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Controllable Neural Story Plot Generation via Reward Shaping](https://mlsanity.com/paper/1809.10736/) (2018)
- [BanditSum: Extractive Summarization as a Contextual Bandit](https://mlsanity.com/paper/1809.09672/) (2018)
- [Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention](https://mlsanity.com/paper/1905.12866/) (2019)
- [Structured Fusion Networks for Dialog](https://mlsanity.com/paper/1907.10016/) (2019)
- [Robustness Testing of Language Understanding in Task-Oriented Dialog](https://mlsanity.com/paper/2012.15262/) (2020)
- [Multi-domain Neural Network Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1603.01232/) (2016)
- [Data Augmentation for Training Dialog Models Robust to Speech Recognition Errors](https://mlsanity.com/paper/2006.05635/) (2020)
- [A Context-aware Natural Language Generator for Dialogue Systems](https://mlsanity.com/paper/1608.07076/) (2016)
- [Natural Language Generation by Hierarchical Decoding with Linguistic Patterns](https://mlsanity.com/paper/1808.02747/) (2018)
- [Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks](https://mlsanity.com/paper/1706.00139/) (2017)

## Built on by (most cited first, 2 of 2)

- [Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies](https://mlsanity.com/paper/2403.17497/) (2024)
- [Empowering NLG: Offline Reinforcement Learning for Informal Summarization in Online Domains](https://mlsanity.com/paper/2306.17174/) (2023)

## Abstract (arXiv)

When a natural language generation (NLG) component is implemented in a real-world task-oriented dialogue system, it is necessary to generate not only natural utterances as learned on training data but also utterances adapted to the dialogue environment (e.g., noise from environmental sounds) and the user (e.g., users with low levels of understanding ability). Inspired by recent advances in reinforcement learning (RL) for language generation tasks, we propose ANTOR, a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning. In ANTOR, a natural language understanding (NLU) module, which corresponds to the user's understanding of system utterances, is incorporated into the objective function of RL. If the NLG's intentions are correctly conveyed to the NLU, which understands a system's utterances, the NLG is given a positive reward. We conducted experiments on the MultiWOZ dataset, and we confirmed that ANTOR could generate adaptive utterances against speech recognition errors and the different vocabulary levels of users.