# Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives

arXiv: https://arxiv.org/abs/2402.11051 · HTML: https://mlsanity.com/paper/2402.11051/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [Modeling Naive Psychology of Characters in Simple Commonsense Stories](https://mlsanity.com/paper/1805.06533/) (2018)
- [CharacterChat: Learning towards Conversational AI with Personalized Social Support](https://mlsanity.com/paper/2308.10278/) (2023)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- ["Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding](https://mlsanity.com/paper/2109.05438/) (2021)
- [Digital Life Project: Autonomous 3D Characters with Social Intelligence](https://mlsanity.com/paper/2312.04547/) (2023)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)
- [Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data](https://mlsanity.com/paper/2205.07557/) (2022)
- [Frowning Frodo, Wincing Leia, and a Seriously Great Friendship: Learning to Classify Emotional Relationships of Fictional Characters](https://mlsanity.com/paper/1903.12453/) (2019)
- [Personality Understanding of Fictional Characters during Book Reading](https://mlsanity.com/paper/2305.10156/) (2023)
- [Annotating Character Relationships in Literary Texts](https://mlsanity.com/paper/1512.00728/) (2015)
- [Whodunnit? Crime Drama as a Case for Natural Language Understanding](https://mlsanity.com/paper/1710.11601/) (2017)
- ["where is this relationship going?": Understanding Relationship Trajectories in Narrative Text](https://mlsanity.com/paper/2010.15313/) (2020)
- [Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts](https://mlsanity.com/paper/2210.05581/) (2022)
- [Relation Clustering in Narrative Knowledge Graphs](https://mlsanity.com/paper/2011.13647/) (2020)

## Built on by (most cited first, 14 of 14)

- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works](https://mlsanity.com/paper/2404.12726/) (2024)
- [TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models](https://mlsanity.com/paper/2405.18027/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Games](https://mlsanity.com/paper/2404.17662/) (2024)
- [Agent-as-Judge for Factual Summarization of Long Narratives](https://mlsanity.com/paper/2501.09993/) (2025)
- [TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games](https://mlsanity.com/paper/2505.15712/) (2025)
- [Cascading Large Language Models for Salient Event Graph Generation](https://mlsanity.com/paper/2406.18449/) (2024)
- [A City of Millions: Mapping Literary Social Networks At Scale](https://mlsanity.com/paper/2502.19590/) (2025)
- [CREFT: Sequential Multi-Agent LLM for Character Relation Extraction](https://mlsanity.com/paper/2505.24553/) (2025)
- [Open Role-Playing with Delta-Engines](https://mlsanity.com/paper/2408.05842/) (2024)
- [SymbolicThought: Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding](https://mlsanity.com/paper/2507.04189/) (2025)
- [Scene-Aware Memory Discrimination: Deciding Which Personal Knowledge Stays](https://mlsanity.com/paper/2602.11607/) (2026)
- [Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?](https://mlsanity.com/paper/2605.17694/) (2026)

## Abstract (arXiv)

Existing datasets for narrative understanding often fail to represent the complexity and uncertainty of relationships in real-life social scenarios. To address this gap, we introduce a new benchmark, Conan, designed for extracting and analysing intricate character relation graphs from detective narratives. Specifically, we designed hierarchical relationship categories and manually extracted and annotated role-oriented relationships from the perspectives of various characters, incorporating both public relationships known to most characters and secret ones known to only a few. Our experiments with advanced Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their limitations in inferencing complex relationships and handling longer narratives. The combination of the Conan dataset and our pipeline strategy is geared towards understanding the ability of LLMs to comprehend nuanced relational dynamics in narrative contexts.