# Unified Knowledge Prompt Pre-training for Customer Service Dialogues

arXiv: https://arxiv.org/abs/2208.14652 · HTML: https://mlsanity.com/paper/2208.14652/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 31 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization](https://mlsanity.com/paper/2204.04362/) (2022)
- [DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder](https://mlsanity.com/paper/2109.10480/) (2021)

## Built on by (most cited first, 2 of 2)

- [InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding](https://mlsanity.com/paper/2306.04933/) (2023)
- [Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2210.08873/) (2022)

## Abstract (arXiv)

Dialogue bots have been widely applied in customer service scenarios to provide timely and user-friendly experience. These bots must classify the appropriate domain of a dialogue, understand the intent of users, and generate proper responses. Existing dialogue pre-training models are designed only for several dialogue tasks and ignore weakly-supervised expert knowledge in customer service dialogues. In this paper, we propose a novel unified knowledge prompt pre-training framework, UFA (Unified Model For All Tasks), for customer service dialogues. We formulate all the tasks of customer service dialogues as a unified text-to-text generation task and introduce a knowledge-driven prompt strategy to jointly learn from a mixture of distinct dialogue tasks. We pre-train UFA on a large-scale Chinese customer service corpus collected from practical scenarios and get significant improvements on both natural language understanding (NLU) and natural language generation (NLG) benchmarks.