# Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations

arXiv: https://arxiv.org/abs/1909.00142 · HTML: https://mlsanity.com/paper/1909.00142/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2019.

## Built on by (most cited first, 22 of 22)

- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation](https://mlsanity.com/paper/2108.12960/) (2021)
- [Language modeling via stochastic processes](https://mlsanity.com/paper/2203.11370/) (2022)
- [Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models](https://mlsanity.com/paper/2105.03495/) (2021)
- [Probing Multilingual Language Models for Discourse](https://mlsanity.com/paper/2106.04832/) (2021)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [A Survey on Zero Pronoun Translation](https://mlsanity.com/paper/2305.10196/) (2023)
- [Evaluating Document Coherence Modelling](https://mlsanity.com/paper/2103.10133/) (2021)
- [Evaluation Benchmarks for Spanish Sentence Representations](https://mlsanity.com/paper/2204.07571/) (2022)
- [Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations](https://mlsanity.com/paper/2109.04602/) (2021)
- [On the data requirements of probing](https://mlsanity.com/paper/2202.12801/) (2022)
- [A Novel Computational and Modeling Foundation for Automatic Coherence Assessment](https://mlsanity.com/paper/2310.00598/) (2023)
- [Quantifying the Task-Specific Information in Text-Based Classifications](https://mlsanity.com/paper/2110.08931/) (2021)
- [PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics](https://mlsanity.com/paper/2201.04275/) (2022)
- [Is neural language acquisition similar to natural? A chronological probing study](https://mlsanity.com/paper/2207.00560/) (2022)
- [Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language models](https://mlsanity.com/paper/2503.04232/) (2025)
- [Evaluating Discourse Cohesion in Pre-trained Language Models](https://mlsanity.com/paper/2503.06137/) (2025)
- [Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders](https://mlsanity.com/paper/2104.05928/) (2021)
- [DLUE: Benchmarking Document Language Understanding](https://mlsanity.com/paper/2305.09520/) (2023)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)
- [Scale, Drivers, and Limits of Representational Alignment Across Large Language Models](https://mlsanity.com/paper/2610.14476/) (2026)

## Abstract (arXiv)

Prior work on pretrained sentence embeddings and benchmarks focus on the capabilities of stand-alone sentences. We propose DiscoEval, a test suite of tasks to evaluate whether sentence representations include broader context information. We also propose a variety of training objectives that makes use of natural annotations from Wikipedia to build sentence encoders capable of modeling discourse. We benchmark sentence encoders pretrained with our proposed training objectives, as well as other popular pretrained sentence encoders on DiscoEval and other sentence evaluation tasks. Empirically, we show that these training objectives help to encode different aspects of information in document structures. Moreover, BERT and ELMo demonstrate strong performances over DiscoEval with individual hidden layers showing different characteristics.