# DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations

arXiv: https://arxiv.org/abs/2404.00557 · HTML: https://mlsanity.com/paper/2404.00557/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 31 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems](https://mlsanity.com/paper/1807.11125/) (2018)

## Built on by (most cited first, 4 of 4)

- [Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models](https://mlsanity.com/paper/2406.12182/) (2024)
- [CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants](https://mlsanity.com/paper/2607.22635/) (2026)
- [Generative AI for Critical Infrastructure in Smart Grids: A Unified Framework for Synthetic Data Generation and Anomaly Detection](https://mlsanity.com/paper/2508.08593/) (2025)
- [Discourse Coherence and Response-Guided Context Rewriting for Multi-Party Dialogue Generation](https://mlsanity.com/paper/2604.06784/) (2026)

## Abstract (arXiv)

Language models pre-trained on general text have achieved impressive results in diverse fields. Yet, the distinct linguistic characteristics of task-oriented dialogues (TOD) compared to general text limit the practical utility of existing language models. Current task-oriented dialogue pre-training methods overlook the one-to-many property of conversations, where multiple responses can be appropriate given the same conversation context. In this paper, we propose a novel dialogue pre-training model called DivTOD, which collaborates with LLMs to learn diverse task-oriented dialogue representations. DivTOD guides LLMs in transferring diverse knowledge to smaller models while removing domain knowledge that contradicts task-oriented dialogues. Experiments show that our model outperforms strong TOD baselines on various downstream dialogue tasks and learns the intrinsic diversity of task-oriented dialogues.