# Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

arXiv: https://arxiv.org/abs/2404.12726 · HTML: https://mlsanity.com/paper/2404.12726/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- ["Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding](https://mlsanity.com/paper/2109.05438/) (2021)
- [Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data](https://mlsanity.com/paper/2205.07557/) (2022)
- [LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming](https://mlsanity.com/paper/2306.08401/) (2023)
- [Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models](https://mlsanity.com/paper/2308.13961/) (2023)
- [Personality Understanding of Fictional Characters during Book Reading](https://mlsanity.com/paper/2305.10156/) (2023)
- [Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind](https://mlsanity.com/paper/2211.04684/) (2022)
- [Multi-level Contrastive Learning for Script-based Character Understanding](https://mlsanity.com/paper/2310.13231/) (2023)
- [Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives](https://mlsanity.com/paper/2402.11051/) (2024)

## Built on by (most cited first, 30 of 38)

- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [The Oscars of AI Theater: A Survey on Role-Playing with Language Models](https://mlsanity.com/paper/2407.11484/) (2024)
- [CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds](https://mlsanity.com/paper/2412.05631/) (2024)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation](https://mlsanity.com/paper/2502.09082/) (2025)
- [How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulation](https://mlsanity.com/paper/2502.14642/) (2025)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data](https://mlsanity.com/paper/2406.18921/) (2024)
- [Reasoning Does Not Necessarily Improve Role-Playing Ability](https://mlsanity.com/paper/2502.16940/) (2025)
- [BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation](https://mlsanity.com/paper/2504.14538/) (2025)
- [MINDECHO: Role-Playing Language Agents for Key Opinion Leaders](https://mlsanity.com/paper/2407.05305/) (2024)
- [FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling](https://mlsanity.com/paper/2504.03775/) (2025)
- [RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing](https://mlsanity.com/paper/2507.20352/) (2025)
- [CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization](https://mlsanity.com/paper/2508.09074/) (2025)
- [Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing](https://mlsanity.com/paper/2409.11726/) (2024)
- [PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?](https://mlsanity.com/paper/2508.10014/) (2025)
- [SPeCtrum: A Grounded Framework for Multidimensional Identity Representation in LLM-Based Agent](https://mlsanity.com/paper/2502.08599/) (2025)
- [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025)
- [Thinking Before Speaking: A Role-playing Model with Mindset](https://mlsanity.com/paper/2409.13752/) (2024)
- [Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents](https://mlsanity.com/paper/2503.08193/) (2025)
- [Codifying Character Logic in Role-Playing](https://mlsanity.com/paper/2505.07705/) (2025)
- [Classifying Unreliable Narrators with Large Language Models](https://mlsanity.com/paper/2506.10231/) (2025)
- [Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing](https://mlsanity.com/paper/2409.16913/) (2024)
- [AI Agents Alone Are Not (Yet) Sufficient for Social Simulation](https://mlsanity.com/paper/2603.00113/) (2026)
- [PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency](https://mlsanity.com/paper/2603.25620/) (2026)
- [Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axes](https://mlsanity.com/paper/2601.04716/) (2026)
- [Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends](https://mlsanity.com/paper/2601.10122/) (2026)
- [HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns](https://mlsanity.com/paper/2601.10198/) (2026)
- [CharacterBench: Benchmarking Character Customization of Large Language Models](https://mlsanity.com/paper/2412.11912/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CroSS dataset from literature experts and assess the generated profiles by comparing them with ground truth references and evaluating their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. Resources are available at https://github.com/Joanna0123/character_profiling.