# GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation

arXiv: https://arxiv.org/abs/2306.01966 · HTML: https://mlsanity.com/paper/2306.01966/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [A Sequence-to-Set Network for Nested Named Entity Recognition](https://mlsanity.com/paper/2105.08901/) (2021)
- [What to do about non-standard (or non-canonical) language in NLP](https://mlsanity.com/paper/1608.07836/) (2016)
- [Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples](https://mlsanity.com/paper/1805.06556/) (2018)
- [Generalisation in Named Entity Recognition: A Quantitative Analysis](https://mlsanity.com/paper/1701.02877/) (2017)
- [OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres](https://mlsanity.com/paper/2106.00933/) (2021)
- [DisCoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detection](https://mlsanity.com/paper/2109.09777/) (2021)
- [Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://mlsanity.com/paper/2302.06488/) (2023)
- [Treebanking User-Generated Content: a UD Based Overview of Guidelines, Corpora and Unified Recommendations](https://mlsanity.com/paper/2011.02063/) (2020)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://mlsanity.com/paper/1704.06779/) (2017)
- [All Roads Lead to UD: Converting Stanford and Penn Parses to English Universal Dependencies with Multilayer Annotations](https://mlsanity.com/paper/1909.00522/) (2019)
- [A Cross-Genre Ensemble Approach to Robust Reddit Part of Speech Tagging](https://mlsanity.com/paper/2004.14312/) (2020)
- [WikiGUM: Exhaustive Entity Linking for Wikification in 12 Genres](https://mlsanity.com/paper/2109.07449/) (2021)
- [Out-of-Domain Evaluation of Finnish Dependency Parsing](https://mlsanity.com/paper/2204.10621/) (2022)

## Built on by (most cited first, 19 of 19)

- [eRST: A Signaled Graph Theory of Discourse Relations and Organization](https://mlsanity.com/paper/2403.13560/) (2024)
- [GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction](https://mlsanity.com/paper/2504.10792/) (2025)
- [GDTB: Genre Diverse Data for English Shallow Discourse Parsing across Modalities, Text Types, and Domains](https://mlsanity.com/paper/2411.00491/) (2024)
- [Subjectivity in the Annotation of Bridging Anaphora](https://mlsanity.com/paper/2506.07297/) (2025)
- [AUTALIC: A Dataset for Anti-AUTistic Ableist Language In Context](https://mlsanity.com/paper/2410.16520/) (2024)
- [Bridging Discourse Treebanks with a Unified Rhetorical Structure Parser](https://mlsanity.com/paper/2510.06427/) (2025)
- [What's Hard in English RST Parsing? Predictive Models for Error Analysis](https://mlsanity.com/paper/2309.04940/) (2023)
- [Unifying the Scope of Bridging Anaphora Types in English: Bridging Annotations in ARRAU and GUM](https://mlsanity.com/paper/2410.01170/) (2024)
- [Not Worth Mentioning? A Pilot Study on Salient Proposition Annotation](https://mlsanity.com/paper/2603.27358/) (2026)
- [DeDisCo at the DISRPT 2025 Shared Task: A System for Discourse Relation Classification](https://mlsanity.com/paper/2509.11498/) (2025)
- [GUMBridge: a Corpus for Varieties of Bridging Anaphora](https://mlsanity.com/paper/2512.07134/) (2025)
- [Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2601.15267/) (2026)
- [SPLICE: A Singleton-Enhanced PipeLIne for Coreference REsolution](https://mlsanity.com/paper/2403.17245/) (2024)
- [Unpacking Ambiguity: The Interaction of Polysemous Discourse Markers and Non-DM Signals](https://mlsanity.com/paper/2507.16748/) (2025)
- [What makes an entity salient in discourse?](https://mlsanity.com/paper/2508.16464/) (2025)
- [CLaC at DISRPT 2025: Hierarchical Adapters for Cross-Framework Multi-lingual Discourse Relation Classification](https://mlsanity.com/paper/2509.16903/) (2025)
- [DiscoExplorer: An Open Interface for the Study of Multilingual Discourse Relations](https://mlsanity.com/paper/2605.15304/) (2026)
- [Calibrated Typed Decisions for Entity, Relation, and Event Extraction](https://mlsanity.com/paper/2610.17113/) (2026)
- [Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic](https://mlsanity.com/paper/2610.17538/) (2026)

## Abstract (arXiv)

We present GENTLE, a new mixed-genre English challenge corpus totaling 17K tokens and consisting of 8 unusual text types for out-of domain evaluation: dictionary entries, esports commentaries, legal documents, medical notes, poetry, mathematical proofs, syllabuses, and threat letters. GENTLE is manually annotated for a variety of popular NLP tasks, including syntactic dependency parsing, entity recognition, coreference resolution, and discourse parsing. We evaluate state-of-the-art NLP systems on GENTLE and find severe degradation for at least some genres in their performance on all tasks, which indicates GENTLE's utility as an evaluation dataset for NLP systems.