# Literary Evidence Retrieval via Long-Context Language Models

arXiv: https://arxiv.org/abs/2506.03090 · HTML: https://mlsanity.com/paper/2506.03090/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 10 of 10)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [FABLES: Evaluating faithfulness and content selection in book-length summarization](https://mlsanity.com/paper/2404.01261/) (2024)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [RELIC: Retrieving Evidence for Literary Claims](https://mlsanity.com/paper/2203.10053/) (2022)
- [HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs](https://mlsanity.com/paper/2405.17633/) (2024)
- [Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions](https://mlsanity.com/paper/2402.13551/) (2024)

## Built on by (most cited first, 1 of 1)

- [Computational Representations of Character Significance in Novels](https://mlsanity.com/paper/2601.15508/) (2026)

## Abstract (arXiv)

How well do modern long-context language models understand literary fiction? We explore this question via the task of literary evidence retrieval, repurposing the RELiC dataset of That et al. (2022) to construct a benchmark where the entire text of a primary source (e.g., The Great Gatsby) is provided to an LLM alongside literary criticism with a missing quotation from that work. This setting, in which the model must generate the missing quotation, mirrors the human process of literary analysis by requiring models to perform both global narrative reasoning and close textual examination. We curate a high-quality subset of 292 examples through extensive filtering and human verification. Our experiments show that recent reasoning models, such as Gemini Pro 2.5 can exceed human expert performance (62.5% vs. 50% accuracy). In contrast, the best open-weight model achieves only 29.1% accuracy, highlighting a wide gap in interpretive reasoning between open and closed-weight models. Despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration, signaling open challenges for applying LLMs to literary analysis. We release our dataset and evaluation code to encourage future work in this direction.