Semantic-based Pre-training for Dialogue Understanding
Not on the map. Why?
Abstract (arXiv)
Pre-trained language models have made great progress on dialogue tasks. However, these models are typically trained on surface dialogue text, thus are proven to be weak in understanding the main semantic meaning of a dialogue context. We investigate Abstract Meaning Representation (AMR) as explicit semantic knowledge for pre-training models to capture the core semantic information in dialogues during pre-training. In particular, we propose a semantic-based pre-training framework that extends the standard pre-training framework (Devlin et al., 2019) by three tasks for learning 1) core semantic units, 2) semantic relations and 3) the overall semantic representation according to AMR graphs. Experiments on the understanding of both chit-chats and task-oriented dialogues show the superiority of our model. To our knowledge, we are the first to leverage a deep semantic representation for dialogue pre-training.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 19 Sep 2022.
Builds on
58 of 78 references.
References · 58
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018Deep contextualized word representationscited by 2,285
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
38 more
- 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
- 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
- 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
- 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
- 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
- 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
- 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
- 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
- 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
- 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
- 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
- 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
- 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
- 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
- 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
- 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
- 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
- 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
- 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
- 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
- 2020Dialogue-Based Relation Extractioncited by 68
- 2018Did the Model Understand the Question?cited by 60
- 2017Neural AMR: Sequence-to-Sequence Models for Parsing and Generationcited by 56
- 2018A Graph-to-Sequence Model for AMR-to-Text Generationcited by 56
- 2022Graph Pre-training for AMR Parsing and Generationcited by 55
- 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48
- 2020AMR Parsing via Graph-Sequence Iterative Inferencecited by 47
- 2019A Repository of Conversational Datasetscited by 45
- 2020Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representationscited by 42
- 2020Conversational Semantic Parsing for Dialog State Trackingcited by 40
- 2020Conversational Semantic Parsingcited by 39
- 2018AMR Parsing as Graph Prediction with Latent Alignmentcited by 36
- 2021MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understandingcited by 36
- 2019Modeling Graph Structure in Transformer for Better AMR-to-Text Generationcited by 35
- 2019Broad-Coverage Semantic Parsing as Transductioncited by 32
- 2020GDPNet: Refining Latent Multi-View Graph for Relation Extractioncited by 29
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.