# AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2102.05126 · HTML: https://mlsanity.com/paper/2102.05126/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training](https://mlsanity.com/paper/1911.03860/) (2019)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Overview of the Ninth Dialog System Technology Challenge: DSTC9](https://mlsanity.com/paper/2011.06486/) (2020)
- [A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning](https://mlsanity.com/paper/2009.08115/) (2020)
- [Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog](https://mlsanity.com/paper/2004.11019/) (2020)
- [Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering](https://mlsanity.com/paper/1906.04229/) (2019)
- [LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization](https://mlsanity.com/paper/2011.09378/) (2020)
- [Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking](https://mlsanity.com/paper/1508.01755/) (2015)
- [Improving Robustness of Task Oriented Dialog Systems](https://mlsanity.com/paper/1911.05153/) (2019)
- [ELITR Non-Native Speech Translation at IWSLT 2020](https://mlsanity.com/paper/2006.03331/) (2020)
- [Shape of synth to come: Why we should use synthetic data for English surface realization](https://mlsanity.com/paper/2005.02693/) (2020)

## Built on by (most cited first, 30 of 31)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [PLACES: Prompting Language Models for Social Conversation Synthesis](https://mlsanity.com/paper/2302.03269/) (2023)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding](https://mlsanity.com/paper/2210.14169/) (2022)
- [Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2501.18845/) (2025)
- [Towards Robust Personalized Dialogue Generation via Order-Insensitive Representation Regularization](https://mlsanity.com/paper/2305.12782/) (2023)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset](https://mlsanity.com/paper/2403.04460/) (2024)
- [Efficient Task-Oriented Dialogue Systems with Response Selection as an Auxiliary Task](https://mlsanity.com/paper/2208.07097/) (2022)
- [Are LLMs Robust for Spoken Dialogues?](https://mlsanity.com/paper/2401.02297/) (2024)
- [Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue](https://mlsanity.com/paper/2406.06399/) (2024)
- [PSYDIAL: Personality-based Synthetic Dialogue Generation using Large Language Models](https://mlsanity.com/paper/2404.00930/) (2024)
- [Dialogue Term Extraction using Transfer Learning and Topological Data Analysis](https://mlsanity.com/paper/2208.10448/) (2022)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems](https://mlsanity.com/paper/2407.12016/) (2024)
- [End-to-End Task-Oriented Dialog Modeling with Semi-Structured Knowledge Management](https://mlsanity.com/paper/2106.11796/) (2021)
- [Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?](https://mlsanity.com/paper/2210.12942/) (2022)
- [Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model](https://mlsanity.com/paper/2207.06839/) (2022)
- [UBARv2: Towards Mitigating Exposure Bias in Task-Oriented Dialogs](https://mlsanity.com/paper/2209.07239/) (2022)
- [LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems](https://mlsanity.com/paper/2311.09390/) (2023)
- [TA&AT: Enhancing Task-Oriented Dialog with Turn-Level Auxiliary Tasks and Action-Tree Based Scheduled Sampling](https://mlsanity.com/paper/2401.15626/) (2024)
- [Towards Better RL Training Data Utilization via Second-Order Rollout](https://mlsanity.com/paper/2602.22765/) (2026)
- [Understanding Emotion Valence is a Joint Deep Learning Task](https://mlsanity.com/paper/2305.17422/) (2023)
- [Curricular Transfer Learning for Sentence Encoded Tasks](https://mlsanity.com/paper/2308.01849/) (2023)
- [Lexical Entrainment for Conversational Systems](https://mlsanity.com/paper/2310.09651/) (2023)
- [Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2406.06127/) (2024)
- [MonoTODia: Translating Monologue Requests to Task-Oriented Dialogues](https://mlsanity.com/paper/2502.17268/) (2025)
- [Collaborative and Proactive Management of Task-Oriented Conversations](https://mlsanity.com/paper/2510.05110/) (2025)
- [LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning](https://mlsanity.com/paper/2602.23610/) (2026)

## Abstract (arXiv)

Attention-based pre-trained language models such as GPT-2 brought considerable progress to end-to-end dialogue modelling. However, they also present considerable risks for task-oriented dialogue, such as lack of knowledge grounding or diversity. To address these issues, we introduce modified training objectives for language model finetuning, and we employ massive data augmentation via back-translation to increase the diversity of the training data. We further examine the possibilities of combining data from multiples sources to improve performance on the target dataset. We carefully evaluate our contributions with both human and automatic methods. Our model substantially outperforms the baseline on the MultiWOZ data and shows competitive performance with state of the art in both automatic and human evaluation.