# Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark

arXiv: https://arxiv.org/abs/2408.16131 · HTML: https://mlsanity.com/paper/2408.16131/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- [Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data](https://mlsanity.com/paper/2205.07557/) (2022)
- [Frowning Frodo, Wincing Leia, and a Seriously Great Friendship: Learning to Classify Emotional Relationships of Fictional Characters](https://mlsanity.com/paper/1903.12453/) (2019)
- [Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism](https://mlsanity.com/paper/1904.08386/) (2019)

## Built on by (most cited first, 3 of 3)

- [NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://mlsanity.com/paper/2510.09869/) (2025)
- [Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey](https://mlsanity.com/paper/2602.15851/) (2026)
- [GraphLit: Learning Text-Enriched Dynamic Character Network Representations for Literary Study](https://mlsanity.com/paper/2605.28643/) (2026)

## Abstract (arXiv)

Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.