# Says Who? Effective Zero-Shot Annotation of Focalization

arXiv: https://arxiv.org/abs/2409.11390 · HTML: https://mlsanity.com/paper/2409.11390/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 61 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 12 of 12)

- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data](https://mlsanity.com/paper/2205.07557/) (2022)
- [Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Tasks](https://mlsanity.com/paper/2305.08714/) (2023)
- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement](https://mlsanity.com/paper/2310.18440/) (2023)
- [Where Do People Tell Stories Online? Story Detection Across Online Communities](https://mlsanity.com/paper/2311.09675/) (2023)
- [Grounding Characters and Places in Narrative Texts](https://mlsanity.com/paper/2305.17561/) (2023)
- [[Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Literary Coreference Annotation with LLMs](https://mlsanity.com/paper/2401.17922/) (2024)
- [Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction](https://mlsanity.com/paper/2303.14342/) (2023)
- [Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models](https://mlsanity.com/paper/2407.10645/) (2024)

## Built on by (most cited first, 2 of 2)

- [Are You There God? Lightweight Narrative Annotation of Christian Fiction with LMs](https://mlsanity.com/paper/2507.19756/) (2025)
- [MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring](https://mlsanity.com/paper/2606.06754/) (2026)

## Abstract (arXiv)

Focalization describes the way in which access to narrative information is restricted or controlled based on the knowledge available to knowledge of the narrator. It is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Even trained annotators frequently disagree on correct labels, suggesting this task is both qualitatively and computationally challenging. In this work, we test how well five contemporary large language model (LLM) families and two baselines perform when annotating short literary excerpts for focalization. Despite the challenging nature of the task, we find that LLMs show comparable performance to trained human annotators, with GPT-4o achieving an average F1 of 84.79%. Further, we demonstrate that the log probabilities output by GPT-family models frequently reflect the difficulty of annotating particular excerpts. Finally, we provide a case study analyzing sixteen Stephen King novels, demonstrating the usefulness of this approach for computational literary studies and the insights gleaned from examining focalization at scale.