mlsanityData to 14 Oct 2026

Structural Pre-training for Dialogue Comprehension

arXiv 2105.10956 ↗First posted 23 May 2021cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PrLMs) have demonstrated superior performance due to their strong ability to learn universal language representations from self-supervised pre-training. However, even with the help of the powerful PrLMs, it is still challenging to effectively capture task-related knowledge from dialogue texts which are enriched by correlations among speaker-aware utterances. In this work, we present SPIDER, Structural Pre-traIned DialoguE Reader, to capture dialogue exclusive features. To simulate the dialogue-like features, we propose two training objectives in addition to the original LM objectives: 1) utterance order restoration, which predicts the order of the permuted utterances in dialogue context; 2) sentence backbone regularization, which regularizes the model to improve the factual correctness of summarized subject-verb-object triplets. Experimental results on widely used dialogue benchmarks verify the effectiveness of the newly introduced self-supervised tasks.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 23 May 2021.

Builds on

43 of 63 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  8. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  10. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  11. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  12. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
31 more
  1. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  2. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  3. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  4. 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 138
  5. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 115
  6. 2019Semantics-aware BERT for Language Understandingcited by 112
  7. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  8. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  9. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  10. 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
  11. 2020Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbotscited by 73
  12. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  13. 2018Modeling Multi-turn Conversation with Deep Utterance Aggregationcited by 65
  14. 2020Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structurecited by 60
  15. 2020Topic-Aware Multi-turn Dialogue Modelingcited by 46
  16. 2020Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selectioncited by 36
  17. 2020DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterancescited by 36
  18. 2019Pretraining Methods for Dialog Context Representation Learningcited by 35
  19. 2020Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialoguescited by 34
  20. 2018FlowQA: Grasping Flow in History for Conversational Machine Comprehensioncited by 33
  21. 2018SDNet: Contextualized Attention-based Deep Network for Conversational Question Answeringcited by 30
  22. 2020Syntax-Enhanced Pre-trained Modelcited by 30
  23. 2018AliMe Assist: An Intelligent Assistant for Creating an Innovative E-commerce Experiencecited by 26
  24. 2019Dual Co-Matching Network for Multi-choice Reading Comprehensioncited by 26
  25. 2019Deep Attentive Ranking Networks for Learning to Order Sentencescited by 22
  26. 2018Deep Enhanced Representation for Implicit Discourse Relation Recognitioncited by 21
  27. 2017Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classificationcited by 17
  28. 2019Parsing All: Syntax and Semantics, Dependencies and Spanscited by 13
  29. 2021Text Compression-aided Transformer Encodingcited by 13
  30. 2020Bipartite Flat-Graph Network for Nested Named Entity Recognitioncited by 11
  31. 2018Lingke: A Fine-grained Multi-turn Chatbot for Customer Servicecited by 10

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 11; 2023: 7.

20 papers.

Sort:

By area:

  1. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  2. 2023Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Modelscited by 11
  3. 2021Structural Characterization for Dialogue Disentanglementcited by 7
  4. 2023Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarizationcited by 7
  5. 2022A Systematic Evaluation of Response Selection for Open Domain Dialoguecited by 4
  6. 2023Pre-training Multi-party Dialogue Models with Latent Discourse Inferencecited by 4
  7. 2023OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue Systemcited by 4
  8. 2022Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehensioncited by 3
  9. 2022Semantic-based Pre-training for Dialogue Understandingcited by 3
  10. 2022Two-Level Supervised Contrastive Learning for Response Selection in Multi-Turn Dialoguecited by 2
  11. 2022Unified Knowledge Prompt Pre-training for Customer Service Dialoguescited by 2
  12. 2023Matching-based Term Semantics Pre-training for Spoken Patient Query Understandingcited by 2
8 more
  1. 2022Language Model Pre-training on True Negativescited by 1
  2. 2023Improve Retrieval-based Dialogue System via Syntax-Informed Attentioncited by 1
  3. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systemscited by 1
  4. 2021Advances in Multi-turn Dialogue Comprehension: A Surveycited by 0
  5. 2022Zero-shot Cross-lingual Conversational Semantic Role Labelingcited by 0
  6. 2022Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialoguescited by 0
  7. 2022DSTEA: Improving Dialogue State Tracking via Entity Adaptive Pre-trainingcited by 0
  8. 2022STRUDEL: Structured Dialogue Summarization for Dialogue Comprehensioncited by 0