# Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systems

arXiv: https://arxiv.org/abs/2210.08873 · HTML: https://mlsanity.com/paper/2210.08873/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Dialog State Tracking: A Neural Reading Comprehension Approach](https://mlsanity.com/paper/1908.01946/) (2019)
- [A Challenge on Semi-Supervised and Reinforced Task-Oriented Dialog Systems](https://mlsanity.com/paper/2207.02657/) (2022)
- [Information Extraction and Human-Robot Dialogue towards Real-life Tasks: A Baseline Study with the MobileCS Dataset](https://mlsanity.com/paper/2209.13464/) (2022)
- [Unified Knowledge Prompt Pre-training for Customer Service Dialogues](https://mlsanity.com/paper/2208.14652/) (2022)

## Built on by (most cited first, 15 of 15)

- [Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.06504/) (2023)
- [Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA](https://mlsanity.com/paper/2308.14436/) (2023)
- [Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT](https://mlsanity.com/paper/2310.10176/) (2023)
- [PreAct: Prediction Enhances Agent's Planning Ability](https://mlsanity.com/paper/2402.11534/) (2024)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [Knowledge-Retrieval Task-Oriented Dialog Systems with Semi-Supervision](https://mlsanity.com/paper/2305.13199/) (2023)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition](https://mlsanity.com/paper/2310.10184/) (2023)
- [Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking](https://mlsanity.com/paper/2310.10520/) (2023)
- [DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations](https://mlsanity.com/paper/2404.00557/) (2024)
- [Watch the Speakers: A Hybrid Continuous Attribution Network for Emotion Recognition in Conversation With Emotion Disentanglement](https://mlsanity.com/paper/2309.09799/) (2023)
- [Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task](https://mlsanity.com/paper/2402.14494/) (2024)
- [BootTOD: Bootstrap Task-oriented Dialogue Representations by Aligning Diverse Responses](https://mlsanity.com/paper/2403.01163/) (2024)
- [Entriever: Energy-based Retriever for Knowledge-Grounded Dialog Systems](https://mlsanity.com/paper/2506.00585/) (2025)
- [Extracting Training Dialogue Data from Large Language Model based Task Bots](https://mlsanity.com/paper/2603.01550/) (2026)

## Abstract (arXiv)

Recent advances in neural approaches greatly improve task-oriented dialogue (TOD) systems which assist users to accomplish their goals. However, such systems rely on costly manually labeled dialogs which are not available in practical scenarios. In this paper, we present our models for Track 2 of the SereTOD 2022 challenge, which is the first challenge of building semi-supervised and reinforced TOD systems on a large-scale real-world Chinese TOD dataset MobileCS. We build a knowledge-grounded dialog model to formulate dialog history and local KB as input and predict the system response. And we perform semi-supervised pre-training both on the labeled and unlabeled data. Our system achieves the first place both in the automatic evaluation and human interaction, especially with higher BLEU (+7.64) and Success (+13.6%) than the second place.