# Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3

arXiv: https://arxiv.org/abs/2406.11380 · HTML: https://mlsanity.com/paper/2406.11380/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data](https://mlsanity.com/paper/2205.07557/) (2022)
- [Personality Understanding of Fictional Characters during Book Reading](https://mlsanity.com/paper/2305.10156/) (2023)
- [Measuring Information Propagation in Literary Social Networks](https://mlsanity.com/paper/2004.13980/) (2020)
- [The Project Dialogism Novel Corpus: A Dataset for Quotation Attribution in Literary Texts](https://mlsanity.com/paper/2204.05836/) (2022)
- [Grounding Characters and Places in Narrative Texts](https://mlsanity.com/paper/2305.17561/) (2023)
- [Improving Automatic Quotation Attribution in Literary Novels](https://mlsanity.com/paper/2307.03734/) (2023)
- [SIG: Speaker Identification in Literature via Prompt-Based Generation](https://mlsanity.com/paper/2312.14590/) (2023)

## Built on by (most cited first, 8 of 8)

- [Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry](https://mlsanity.com/paper/2409.00060/) (2024)
- [Analyzing Nobel Prize Literature with Large Language Models](https://mlsanity.com/paper/2410.18142/) (2024)
- [Are You There God? Lightweight Narrative Annotation of Christian Fiction with LMs](https://mlsanity.com/paper/2507.19756/) (2025)
- [Computational Narrative Understanding for Expressive Text-to-Speech](https://mlsanity.com/paper/2509.04072/) (2025)
- [Text-to-Stage: Spatial Layouts from Long-form Narratives](https://mlsanity.com/paper/2603.17832/) (2026)
- [Attribution Bias in Large Language Models](https://mlsanity.com/paper/2604.05224/) (2026)
- [Reinforcing privacy reasoning in LLMs via normative simulacra from fiction](https://mlsanity.com/paper/2604.20904/) (2026)
- [Fast and Accurate Quotation Attribution in Literary Texts](https://mlsanity.com/paper/2608.02359/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination. We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data.