# DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking

arXiv: https://arxiv.org/abs/2510.17013 · HTML: https://mlsanity.com/paper/2510.17013/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 46 references that are in the map.
- First posted on arXiv 19 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model](https://mlsanity.com/paper/2003.01355/) (2020)
- [Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks](https://mlsanity.com/paper/2405.01719/) (2024)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres](https://mlsanity.com/paper/2401.17974/) (2024)
- [Implicit Discourse Relation Classification For Nigerian Pidgin](https://mlsanity.com/paper/2406.18776/) (2024)
- [GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction](https://mlsanity.com/paper/2504.10792/) (2025)
- [Subjectivity in the Annotation of Bridging Anaphora](https://mlsanity.com/paper/2506.07297/) (2025)

## Built on by (most cited first, 2 of 2)

- [BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models](https://mlsanity.com/paper/2511.13095/) (2025)
- [LLMBridge: An LLM Pipeline for End-to-end Referential Bridging Resolution in English](https://mlsanity.com/paper/2605.29048/) (2026)

## Abstract (arXiv)

Recent LLM benchmarks have tested models on a range of phenomena, but are still focused primarily on natural language understanding for extraction of explicit information, such as QA or summarization, with responses often targeting information from individual sentences. We are still lacking more challenging, and importantly also multilingual, benchmarks focusing on implicit information and pragmatic inferences across larger documents in the context of discourse tracking: integrating and aggregating information across sentences, paragraphs and multiple speaker utterances. To this end, we present DiscoTrack, an LLM benchmark targeting a range of tasks across 12 languages and four levels of discourse understanding: salience recognition, entity tracking, discourse relations and bridging inference. Our evaluation shows that these tasks remain challenging, even for state-of-the-art models.