mlsanityData to 7 Oct 2026

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

arXiv 2004.06871 ↗First posted 15 Apr 2020cs.CLcited by 185

Not on the map. Why?

Abstract (arXiv)

The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 59; 2023: 39; 2024: 18; 2025: 12; 2026: 9.

185 papers.

Sort:

By area:

  1. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  4. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  5. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  6. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  7. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  8. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  9. 2021Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Trackingcited by 61
  10. 2021Zero-Shot Dialogue State Tracking via Cross-Task Transfercited by 52
  11. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  12. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
38 more
  1. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  2. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  3. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  4. 2021AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Modelscited by 31
  5. 2021CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systemscited by 30
  6. 2022Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialogcited by 30
  7. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  8. 2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
  9. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  10. 2021Slot Self-Attentive Dialogue State Trackingcited by 27
  11. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  12. 2021Effectiveness of Pre-training for Few-shot Intent Classificationcited by 27
  13. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  14. 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
  15. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  16. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  17. 2021DS-TOD: Efficient Domain Specialization for Task Oriented Dialogcited by 21
  18. 2021NER-BERT: A Pre-trained Model for Low-Resource Entity Taggingcited by 21
  19. 2022Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Trackingcited by 21
  20. 2021Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Trackingcited by 20
  21. 2021ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialoguescited by 20
  22. 2022BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialogcited by 20
  23. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  24. 2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systemscited by 18
  25. 2021QAConv: Question Answering on Informative Conversationscited by 17
  26. 2022Learning Dialogue Representations from Consecutive Utterancescited by 17
  27. 2023Towards Robust Personalized Dialogue Generation via Order-Insensitive Representation Regularizationcited by 17
  28. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Trackingcited by 17
  29. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 16
  30. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  31. 2022Robust Dialogue State Tracking with Weak Supervision and Sparse Datacited by 15
  32. 2022Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systemscited by 15
  33. 2022A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialogcited by 14
  34. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  35. 2021Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representationscited by 13
  36. 2021Advances in Multi-turn Dialogue Comprehension: A Surveycited by 12
  37. 2021Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understandingcited by 12
  38. 2021Response Generation with Context-Aware Prompt Learningcited by 12

50 most cited of 185 All 185 (JSON)