mlsanityData to 7 Oct 2026

A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

arXiv 2307.03378 ↗First posted 7 Jul 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification. This hinders researchers from fully utilizing public-available models in discourse analysis. This work is a straightforward, fine-tuned discourse performance comparison of seven pre-trained language models. We use PDTB-3, a popular discourse relation annotated dataset. Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations. Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best performing model for implicit discourse relation classification. Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

23 of 51 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
11 more
  1. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2020Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Modelscited by 49
  4. 2018Deep Enhanced Representation for Implicit Discourse Relation Recognitioncited by 21
  5. 2017Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classificationcited by 17
  6. 2018Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraphcited by 17
  7. 2018Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classificationcited by 16
  8. 2021Probing Multilingual Language Models for Discoursecited by 12
  9. 2021Let's be explicit about that: Distant supervision for implicit discourse relation classification via connective predictioncited by 9
  10. 2022Revisiting Shallow Discourse Parsing in the PDTB-3: Handling Intra-sentential Implicitscited by 4
  11. 2019Memorizing All for Implicit Discourse Relation Recognitioncited by 3

Built on by

Not cited yet.