# Less is More: A Lightweight and Robust Neural Architecture for Discourse Parsing

arXiv: https://arxiv.org/abs/2210.09537 · HTML: https://mlsanity.com/paper/2210.09537/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 80 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Towards Understanding Regularization in Batch Normalization](https://mlsanity.com/paper/1809.00846/) (2018)
- [On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification](https://mlsanity.com/paper/2004.12617/) (2020)
- [Deep Enhanced Representation for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/1807.05154/) (2018)
- [Cross-lingual RST Discourse Parsing](https://mlsanity.com/paper/1701.02946/) (2017)
- [Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph](https://mlsanity.com/paper/1804.05918/) (2018)
- [Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1811.01697/) (2018)
- [A Top-Down Neural Architecture towards Text-Level Parsing of Discourse Rhetorical Structure](https://mlsanity.com/paper/2005.02680/) (2020)
- [Top-down Discourse Parsing via Sequence Labelling](https://mlsanity.com/paper/2102.02080/) (2021)
- [A Latent Variable Recurrent Neural Network for Discourse Relation Language Models](https://mlsanity.com/paper/1603.01913/) (2016)
- [Neural Generative Rhetorical Structure Parsing](https://mlsanity.com/paper/1909.11049/) (2019)
- [Memorizing All for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/1908.11317/) (2019)

## Built on by (most cited first, 2 of 2)

- [RST-style Discourse Parsing Guided by Document-level Content Structures](https://mlsanity.com/paper/2309.04141/) (2023)
- [Semi-supervised News Discourse Profiling with Contrastive Learning](https://mlsanity.com/paper/2309.11692/) (2023)

## Abstract (arXiv)

Complex feature extractors are widely employed for text representation building. However, these complex feature extractors make the NLP systems prone to overfitting especially when the downstream training datasets are relatively small, which is the case for several discourse parsing tasks. Thus, we propose an alternative lightweight neural architecture that removes multiple complex feature extractors and only utilizes learnable self-attention modules to indirectly exploit pretrained neural language models, in order to maximally preserve the generalizability of pre-trained language models. Experiments on three common discourse parsing tasks show that powered by recent pretrained language models, the lightweight architecture consisting of only two self-attention layers obtains much better generalizability and robustness. Meanwhile, it achieves comparable or even better system performance with fewer learnable parameters and less processing time.