mlsanityData to 7 Oct 2026

Dialogue-oriented Pre-training

arXiv 2106.00420 ↗First posted 1 Jun 2021cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PrLM) has been shown powerful in enhancing a broad range of downstream tasks including various dialogue related ones. However, PrLMs are usually trained on general plain text with common language model (LM) training objectives, which cannot sufficiently capture dialogue exclusive features due to the limitation of such training setting, so that there is an immediate need to fill the gap between a specific dialogue task and the LM task. As it is unlikely to collect huge dialogue data for dialogue-oriented pre-training, in this paper, we propose three strategies to simulate the conversation features on general plain text. Our proposed method differs from existing post-training methods that it may yield a general-purpose PrLM and does not individualize to any detailed task while keeping the capability of learning dialogue related features including speaker awareness, continuity and consistency. The resulted Dialog-PrLM is fine-tuned on three public multi-turn dialogue datasets and helps achieve significant and consistent improvement over the plain PrLMs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 1 Jun 2021.

Builds on

33 of 47 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  8. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  9. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  10. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  11. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  12. 2019Semantics-aware BERT for Language Understandingcited by 111
21 more
  1. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  2. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  3. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  4. 2020Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbotscited by 73
  5. 2018Modeling Multi-turn Conversation with Deep Utterance Aggregationcited by 65
  6. 2015Learning Natural Language Inference with LSTMcited by 51
  7. 2020Topic-Aware Multi-turn Dialogue Modelingcited by 46
  8. 2020Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selectioncited by 36
  9. 2020Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialoguescited by 34
  10. 2016Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNNcited by 27
  11. 2019Dual Co-Matching Network for Multi-choice Reading Comprehensioncited by 26
  12. 2020Response Selection for Multi-Party Conversations with Dynamic Topic Trackingcited by 26
  13. 2020Dialogue Graph Modeling for Conversational Machine Readingcited by 26
  14. 2020Multi-turn Response Selection using Dialogue Dependency Relationscited by 23
  15. 2015LSTM-based Deep Learning Models for Non-factoid Answer Selectioncited by 17
  16. 2019Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbotscited by 13
  17. 2021Text Compression-aided Transformer Encodingcited by 13
  18. 2015Improved Deep Learning Baselines for Ubuntu Corpus Dialogscited by 12
  19. 2021Advances in Multi-turn Dialogue Comprehension: A Surveycited by 12
  20. 2018Lingke: A Fine-grained Multi-turn Chatbot for Customer Servicecited by 10
  21. 2019LIMIT-BERT : Linguistic Informed Multi-Task BERTcited by 10

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 4; 2023: 3.

9 papers.

  1. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  2. 2021Enhanced Speaker-aware Multi-party Multi-turn Dialogue Comprehensioncited by 13
  3. 2022Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learningcited by 5
  4. 2023Pre-training Multi-party Dialogue Models with Latent Discourse Inferencecited by 4
  5. 2022DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learningcited by 3
  6. 2022KPT: Keyword-guided Pre-training for Grounded Dialog Generationcited by 3
  7. 2022Instance Regularization for Discriminative Language Model Pre-trainingcited by 2
  8. 2023Matching-based Term Semantics Pre-training for Spoken Patient Query Understandingcited by 2
  9. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systemscited by 1