# Domain-Adaptive Pretraining Methods for Dialogue Understanding

arXiv: https://arxiv.org/abs/2105.13665 · HTML: https://mlsanity.com/paper/2105.13665/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Proactive Human-Machine Conversation with Explicit Conversation Goals](https://mlsanity.com/paper/1906.05572/) (2019)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019)
- [NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven Conversation](https://mlsanity.com/paper/2103.02548/) (2021)
- [Semantic Role Labeling Guided Multi-turn Dialogue ReWriter](https://mlsanity.com/paper/2010.01417/) (2020)
- [Conversational Semantic Role Labeling](https://mlsanity.com/paper/2104.04947/) (2021)

## Built on by (most cited first, 11 of 11)

- [Representation Surgery: Theory and Practice of Affine Steering](https://mlsanity.com/paper/2402.09631/) (2024)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessment](https://mlsanity.com/paper/2311.07603/) (2023)
- [Conversational Semantic Role Labeling with Predicate-Oriented Latent Graph](https://mlsanity.com/paper/2210.03037/) (2022)
- [Task-guided Disentangled Tuning for Pretrained Language Models](https://mlsanity.com/paper/2203.11431/) (2022)
- [DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization](https://mlsanity.com/paper/2406.04669/) (2024)
- [A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generation](https://mlsanity.com/paper/2406.09881/) (2024)
- [TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining](https://mlsanity.com/paper/2410.10006/) (2024)
- [QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Prediction](https://mlsanity.com/paper/2505.07863/) (2025)
- [LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Data](https://mlsanity.com/paper/2506.14474/) (2025)
- [Zero-shot Cross-lingual Conversational Semantic Role Labeling](https://mlsanity.com/paper/2204.04914/) (2022)

## Abstract (arXiv)

Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.