mlsanityData to 7 Oct 2026

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

arXiv 2209.04595 ↗First posted 10 Sep 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user's constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and get competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 10 Sep 2022.

Builds on

57 of 74 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  5. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  6. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  7. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  8. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  9. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  10. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  11. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  12. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
38 more
  1. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  2. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  3. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  4. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  5. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  6. 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 137
  7. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  8. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  9. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  10. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  11. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  12. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  13. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  14. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  15. 2019Efficient Dialogue State Tracking by Selectively Overwriting Memorycited by 76
  16. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  17. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  18. 2019SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Trackingcited by 63
  19. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  20. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  21. 2018Global-Locally Self-Attentive Dialogue State Trackercited by 45
  22. 2020CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackerscited by 43
  23. 2019Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answeringcited by 39
  24. 2019Pretraining Methods for Dialog Context Representation Learningcited by 35
  25. 2021Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Trackingcited by 35
  26. 2018Towards Universal Dialogue State Trackingcited by 33
  27. 2019ConvLab: Multi-Domain End-to-End Dialog System Platformcited by 29
  28. 2021SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Miningcited by 24
  29. 2018Zero-Shot Dialog Generation with Cross-Domain Latent Actionscited by 23
  30. 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
  31. 2019Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generationcited by 21
  32. 2016Conditional Generation and Snapshot Learning in Neural Dialogue Systemscited by 17
  33. 2018Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spacescited by 15
  34. 2020Non-Autoregressive Dialog State Trackingcited by 15
  35. 2019HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Trackingcited by 14
  36. 2020Hierarchical Transformer for Task Oriented Dialog Systemscited by 14
  37. 2019AgentGraph: Towards Universal Dialogue Management with Structured Deep Reinforcement Learningcited by 12
  38. 2020Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Managementcited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2024MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systemscited by 7
  2. 2025Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Modelscited by 2
  3. 2023Document Automation Architectures: Updated Survey in Light of Large Language Modelscited by 1
  4. 2023A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-trainingcited by 1