# Task-Oriented Dialogue System as Natural Language Generation

arXiv: https://arxiv.org/abs/2108.13679 · HTML: https://mlsanity.com/paper/2108.13679/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2](https://mlsanity.com/paper/2012.03539/) (2020)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Efficient Dialogue State Tracking by Selectively Overwriting Memory](https://mlsanity.com/paper/1911.03906/) (2019)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching](https://mlsanity.com/paper/2005.05298/) (2020)
- [Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/1804.08217/) (2018)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention](https://mlsanity.com/paper/1905.12866/) (2019)
- [Structured Fusion Networks for Dialog](https://mlsanity.com/paper/1907.10016/) (2019)
- [Incorporating BERT into Parallel Sequence Decoding with Adapters](https://mlsanity.com/paper/2010.06138/) (2020)
- [ConvLab: Multi-Domain End-to-End Dialog System Platform](https://mlsanity.com/paper/1904.08637/) (2019)
- [The Eighth Dialog System Technology Challenge](https://mlsanity.com/paper/1911.06394/) (2019)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking](https://mlsanity.com/paper/2104.04466/) (2021)
- [UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues](https://mlsanity.com/paper/2004.14307/) (2020)
- [Non-Autoregressive Dialog State Tracking](https://mlsanity.com/paper/2002.08024/) (2020)
- [HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking](https://mlsanity.com/paper/1907.00883/) (2019)
- [OneNet: Joint Domain, Intent, Slot Prediction for Spoken Language Understanding](https://mlsanity.com/paper/1801.05149/) (2018)
- [Budgeted Policy Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1906.00499/) (2019)

## Built on by (most cited first, 14 of 14)

- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation](https://mlsanity.com/paper/2310.07968/) (2023)
- [Bot or Human? Detecting ChatGPT Imposters with A Single Question](https://mlsanity.com/paper/2305.06424/) (2023)
- [Visually-Augmented Language Modeling](https://mlsanity.com/paper/2205.10178/) (2022)
- [Automatic Song Translation for Tonal Languages](https://mlsanity.com/paper/2203.13420/) (2022)
- [CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialogue](https://mlsanity.com/paper/2303.11108/) (2023)
- [Improved and Efficient Conversational Slot Labeling through Question Answering](https://mlsanity.com/paper/2204.02123/) (2022)
- [GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2405.11180/) (2024)
- [STEPS: A Benchmark for Order Reasoning in Sequential Tasks](https://mlsanity.com/paper/2306.04441/) (2023)
- [AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling Agents](https://mlsanity.com/paper/2601.02643/) (2026)
- [GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant](https://mlsanity.com/paper/2603.01059/) (2026)
- [IMTLab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems](https://mlsanity.com/paper/2310.11163/) (2023)
- [Leveraging Graph Structures and Large Language Models for End-to-End Synthetic Task-Oriented Dialogues](https://mlsanity.com/paper/2501.11977/) (2025)
- [Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics](https://mlsanity.com/paper/2603.29518/) (2026)

## Abstract (arXiv)

In this paper, we propose to formulate the task-oriented dialogue system as the purely natural language generation task, so as to fully leverage the large-scale pre-trained models like GPT-2 and simplify complicated delexicalization prepossessing. However, directly applying this method heavily suffers from the dialogue entity inconsistency caused by the removal of delexicalized tokens, as well as the catastrophic forgetting problem of the pre-trained model during fine-tuning, leading to unsatisfactory performance. To alleviate these problems, we design a novel GPT-Adapter-CopyNet network, which incorporates the lightweight adapter and CopyNet modules into GPT-2 to achieve better performance on transfer learning and dialogue entity generation. Experimental results conducted on the DSTC8 Track 1 benchmark and MultiWOZ dataset demonstrate that our proposed approach significantly outperforms baseline models with a remarkable performance on automatic and human evaluations.