# MuLD: The Multitask Long Document Benchmark

arXiv: https://arxiv.org/abs/2202.07362 · HTML: https://mlsanity.com/paper/2202.07362/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [Blockwise Self-Attention for Long Document Understanding](https://mlsanity.com/paper/1911.02972/) (2019)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2112.13610/) (2021)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)

## Built on by (most cited first, 9 of 9)

- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Investigating Efficiently Extending Transformers for Long Input Summarization](https://mlsanity.com/paper/2208.04347/) (2022)
- [One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support](https://mlsanity.com/paper/2306.09237/) (2023)
- [Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs](https://mlsanity.com/paper/2509.21361/) (2025)
- [Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks](https://mlsanity.com/paper/2410.08133/) (2024)
- [Single-Pass Document Scanning for Question Answering](https://mlsanity.com/paper/2504.03101/) (2025)
- [NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents](https://mlsanity.com/paper/2402.17682/) (2024)
- [DLUE: Benchmarking Document Language Understanding](https://mlsanity.com/paper/2305.09520/) (2023)
- [From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization](https://mlsanity.com/paper/2603.00021/) (2026)

## Abstract (arXiv)

The impressive progress in NLP techniques has been driven by the development of multi-task benchmarks such as GLUE and SuperGLUE. While these benchmarks focus on tasks for one or two input sentences, there has been exciting work in designing efficient techniques for processing much longer inputs. In this paper, we present MuLD: a new long document benchmark consisting of only documents over 10,000 tokens. By modifying existing NLP tasks, we create a diverse benchmark which requires models to successfully model long-term dependencies in the text. We evaluate how existing models perform, and find that our benchmark is much more challenging than their `short document' equivalents. Furthermore, by evaluating both regular and efficient transformers, we show that models with increased context length are better able to solve the tasks presented, suggesting that future improvements in these models are vital for solving similar long document problems. We release the data and code for baselines to encourage further research on efficient NLP models.