# A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

arXiv: https://arxiv.org/abs/2307.03378 · HTML: https://mlsanity.com/paper/2307.03378/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [Deep Enhanced Representation for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/1807.05154/) (2018)
- [Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1704.00217/) (2017)
- [Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph](https://mlsanity.com/paper/1804.05918/) (2018)
- [Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1811.01697/) (2018)
- [Probing Multilingual Language Models for Discourse](https://mlsanity.com/paper/2106.04832/) (2021)
- [Let's be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction](https://mlsanity.com/paper/2106.03192/) (2021)
- [Revisiting Shallow Discourse Parsing in the PDTB-3: Handling Intra-sentential Implicits](https://mlsanity.com/paper/2204.00350/) (2022)
- [Memorizing All for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/1908.11317/) (2019)

## Abstract (arXiv)

Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification. This hinders researchers from fully utilizing public-available models in discourse analysis. This work is a straightforward, fine-tuned discourse performance comparison of seven pre-trained language models. We use PDTB-3, a popular discourse relation annotated dataset. Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations. Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best performing model for implicit discourse relation classification. Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance.