# The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error

arXiv: https://arxiv.org/abs/2203.11317 · HTML: https://mlsanity.com/paper/2203.11317/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 71 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift](https://mlsanity.com/paper/2003.04475/) (2020)
- [A survey on domain adaptation theory: learning bounds and theoretical guarantees](https://mlsanity.com/paper/2004.11829/) (2020)
- [Support and Invertibility in Domain-Invariant Representations](https://mlsanity.com/paper/1903.03448/) (2019)
- [Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment](https://mlsanity.com/paper/1903.01689/) (2019)
- [Better Document-level Sentiment Analysis from RST Discourse Parsing](https://mlsanity.com/paper/1509.01599/) (2015)
- [Cross-lingual RST Discourse Parsing](https://mlsanity.com/paper/1701.02946/) (2017)
- [Unsupervised Domain Adaptation Based on Source-guided Discrepancy](https://mlsanity.com/paper/1809.03839/) (2018)
- [Discourse Level Factors for Sentence Deletion in Text Simplification](https://mlsanity.com/paper/1911.10384/) (2019)
- [Data Selection Strategies for Multi-Domain Sentiment Analysis](https://mlsanity.com/paper/1702.02426/) (2017)
- [Extending Implicit Discourse Relation Recognition to the PDTB-3](https://mlsanity.com/paper/2010.06294/) (2020)

## Built on by (most cited first, 12 of 12)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [APPDIA: A Discourse-aware Transformer-based Style Transfer Model for Offensive Social Media Conversations](https://mlsanity.com/paper/2209.08207/) (2022)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://mlsanity.com/paper/2302.06488/) (2023)
- [LEATHER: A Framework for Learning to Generate Human-like Text in Dialogue](https://mlsanity.com/paper/2210.07777/) (2022)
- [Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insights](https://mlsanity.com/paper/2207.07255/) (2022)
- [What's Hard in English RST Parsing? Predictive Models for Error Analysis](https://mlsanity.com/paper/2309.04940/) (2023)
- [Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples](https://mlsanity.com/paper/2306.01206/) (2023)
- [End-to-End Argument Mining over Varying Rhetorical Structures](https://mlsanity.com/paper/2401.11218/) (2024)
- [Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2503.20588/) (2025)
- [DISCO: DISCovering Overfittings as Causal Rules for Text Classification Models](https://mlsanity.com/paper/2411.04649/) (2024)

## Abstract (arXiv)

Discourse analysis allows us to attain inferences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside of the training distribution's coverage, diminishing the practical utility of existing models. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these samples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly correlate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our proposal and help us to understand its limitations, but also provide insight on the properties of discourse models and datasets which improve performance in domain adaptation. For instance, we find that non-news datasets are slightly easier to transfer to than news datasets when the training and test sets are very different. Our code and an associated Python package are available to allow practitioners to make more informed model and dataset choices.