# Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring

arXiv: https://arxiv.org/abs/2010.06137 · HTML: https://mlsanity.com/paper/2010.06137/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [LPAttack: A Feasible Annotation Scheme for Capturing Logic Pattern of Attacks in Arguments](https://mlsanity.com/paper/2204.01512/) (2022)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)
- [From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding](https://mlsanity.com/paper/2511.20547/) (2025)
- [Empirical Comparison of Encoder-Based Language Models and Feature-Based Supervised Machine Learning Approaches to Automated Scoring of Long Essays](https://mlsanity.com/paper/2601.02659/) (2026)

## Abstract (arXiv)

Existing approaches for automated essay scoring and document representation learning typically rely on discourse parsers to incorporate discourse structure into text representation. However, the performance of parsers is not always adequate, especially when they are used on noisy texts, such as student essays. In this paper, we propose an unsupervised pre-training approach to capture discourse structure of essays in terms of coherence and cohesion that does not require any discourse parser or annotation. We introduce several types of token, sentence and paragraph-level corruption techniques for our proposed pre-training approach and augment masked language modeling pre-training with our pre-training method to leverage both contextualized and discourse information. Our proposed unsupervised approach achieves new state-of-the-art result on essay Organization scoring task.