mlsanityData to 7 Oct 2026

Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System

arXiv 2109.14739 ↗First posted 29 Sep 2021cs.CLcited by 139

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models have been recently shown to benefit task-oriented dialogue (TOD) systems. Despite their success, existing methods often formulate this task as a cascaded generation problem which can lead to error accumulation across different sub-tasks and greater data annotation overhead. In this study, we present PPTOD, a unified plug-and-play model for task-oriented dialogue. In addition, we introduce a new dialogue multi-task pre-training strategy that allows the model to learn the primary TOD task completion skills from heterogeneous dialog corpora. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification. Experimental results show that PPTOD achieves new state of the art on all evaluated tasks in both high-resource and low-resource scenarios. Furthermore, comparisons against previous SOTA methods show that the responses generated by PPTOD are more factually correct and semantically coherent as judged by human annotators.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Builds on 60 of its 84 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

60 of 84 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  12. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
38 more
  1. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  2. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  3. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  4. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  5. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  6. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  7. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  8. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  9. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  11. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  12. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  13. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  14. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  15. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  16. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  17. 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 137
  18. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  19. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  20. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  21. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  22. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  23. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  24. 2017Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systemscited by 84
  25. 2019Efficient Dialogue State Tracking by Selectively Overwriting Memorycited by 76
  26. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  27. 2019SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Trackingcited by 63
  28. 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
  29. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  30. 2020A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learningcited by 47
  31. 2018Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systemscited by 46
  32. 2018Global-Locally Self-Attentive Dialogue State Trackercited by 45
  33. 2020A Sequence-to-Sequence Approach to Dialogue State Trackingcited by 43
  34. 2019Structured Fusion Networks for Dialogcited by 40
  35. 2019Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answeringcited by 39
  36. 2019The Eighth Dialog System Technology Challengecited by 25
  37. 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
  38. 2020Prototype-to-Style: Dialogue Generation with Style-Aware Editing on Retrieval Memorycited by 22

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 46; 2023: 43; 2024: 28; 2025: 11; 2026: 6.

139 papers.

Sort:

By area:

  1. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  2. 2022A Contrastive Framework for Neural Text Generationcited by 224
  3. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  4. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  5. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  6. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  7. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  8. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  9. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  10. 2023MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphscited by 40
  11. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  12. 2023SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Promptingcited by 39
38 more
  1. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  2. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Callingcited by 34
  3. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  4. 2022A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systemscited by 32
  5. 2023Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue Systemcited by 32
  6. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  7. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  8. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  9. 2021DS-TOD: Efficient Domain Specialization for Task Oriented Dialogcited by 21
  10. 2022Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Trackingcited by 21
  11. 2024MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotationscited by 21
  12. 2025Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Modelcited by 20
  13. 2023Learning to Extrapolate: A Transductive Approachcited by 19
  14. 2023User Simulation with Large Language Models for Evaluating Task-Oriented Dialoguecited by 19
  15. 2022Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Trackingcited by 18
  16. 2023End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directionscited by 17
  17. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 17
  18. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  19. 2023Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systemscited by 16
  20. 2024MT2ST: Adaptive Multi-Task to Single-Task Learningcited by 16
  21. 2022Robust Dialogue State Tracking with Weak Supervision and Sparse Datacited by 15
  22. 2022Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systemscited by 15
  23. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  24. 2025Adaptive Social Learning via Mode Policy Optimization for Language Agentscited by 15
  25. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  26. 2024Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversationscited by 13
  27. 2021UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialoguescited by 12
  28. 2023CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialoguecited by 12
  29. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  30. 2022KETOD: Knowledge-Enriched Task-Oriented Dialoguecited by 11
  31. 2022Mars: Modeling Context & State Representations with Contrastive Learning for End-to-End Task-Oriented Dialogcited by 11
  32. 2022Q-TOD: A Query-driven Task-oriented Dialogue Systemcited by 10
  33. 2023Diverse Retrieval-Augmented In-Context Learning for Dialogue State Trackingcited by 10
  34. 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisonscited by 10
  35. 2021Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnosticscited by 9
  36. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  37. 2022UniDU: Towards A Unified Generative Dialogue Understanding Frameworkcited by 9
  38. 2022KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learningcited by 9

50 most cited of 139 All 139 (JSON)