mlsanityData to 7 Oct 2026

ChapterBreak: A Challenge Dataset for Long-Range Language Models

arXiv 2204.10878 ↗First posted 22 Apr 2022cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce ChapterBreak, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from a set of negative segments sampled from the same narrative. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on ChapterBreak show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our ChapterBreak dataset to spur more principled future research into LRLMs.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

33 of 42 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2020Efficient Transformers: A Surveycited by 784
21 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2014Memory Networkscited by 354
  7. 2021Efficient Attentions for Long Document Summarizationcited by 342
  8. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  9. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  10. 2022Memorizing Transformerscited by 219
  11. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  12. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  13. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  14. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  15. 2022Block-Recurrent Transformerscited by 107
  16. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  17. 2021Adaptive Semiparametric Language Modelscited by 83
  18. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  19. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  20. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  21. 2020Toward Better Storylines with Sentence-Level Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 7; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  3. 2023Augmenting Language Models with Long-Term Memorycited by 186
  4. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  5. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  6. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  7. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  8. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  9. 2022Model Criticism for Long-Form Text Generationcited by 21
  10. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  11. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  12. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 9
6 more
  1. 2022Black-box language model explanation by context length probingcited by 7
  2. 2024Integrating Planning into Single-Turn Long-Form Text Generationcited by 7
  3. 2023STORYWARS: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generationcited by 5
  4. 2022An Inclusive Notion of Textcited by 0
  5. 2025FairFlow: Mitigating Dataset Biases through Undecided Learningcited by 0
  6. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0