mlsanityData to 7 Oct 2026

Domain-Adaptive Pretraining Methods for Dialogue Understanding

arXiv 2105.13665 ↗First posted 28 May 2021cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

15 of 18 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  10. 2019Proactive Human-Machine Conversation with Explicit Conversation Goalscited by 87
  11. 2020CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Datasetcited by 66
  12. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
3 more
  1. 2021NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven Conversationcited by 35
  2. 2020Semantic Role Labeling Guided Multi-turn Dialogue ReWritercited by 15
  3. 2021Conversational Semantic Role Labelingcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 4; 2025: 2.

11 papers.

  1. 2024Representation Surgery: Theory and Practice of Affine Steeringcited by 47
  2. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  3. 2023PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessmentcited by 19
  4. 2022Conversational Semantic Role Labeling with Predicate-Oriented Latent Graphcited by 14
  5. 2022Task-guided Disentangled Tuning for Pretrained Language Modelscited by 1
  6. 2024DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalizationcited by 1
  7. 2024A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generationcited by 1
  8. 2024TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretrainingcited by 1
  9. 2025QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Predictioncited by 1
  10. 2025LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Datacited by 1
  11. 2022Zero-shot Cross-lingual Conversational Semantic Role Labelingcited by 0