# CHARM: Character Hallucination for Multicultural Role Play Benchmark

arXiv: https://arxiv.org/abs/2609.01352 · HTML: https://mlsanity.com/paper/2609.01352/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent](https://mlsanity.com/paper/2402.13717/) (2024)
- [RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems](https://mlsanity.com/paper/2409.16727/) (2024)
- [Mitigating Hallucination in Fictional Character Role-Play](https://mlsanity.com/paper/2406.17260/) (2024)
- [Unsupervised Enrichment of Persona-grounded Dialog with Background Stories](https://mlsanity.com/paper/2106.08364/) (2021)
- [Telling Stories through Multi-User Dialogue by Modeling Character Relations](https://mlsanity.com/paper/2105.15054/) (2021)
- [RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing](https://mlsanity.com/paper/2507.20352/) (2025)
- [Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing](https://mlsanity.com/paper/2409.11726/) (2024)
- [Concept Incongruence: An Exploration of Time and Death in Role Playing](https://mlsanity.com/paper/2505.14905/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Curse of Knowledge in Role-Playing Language Models](https://mlsanity.com/paper/2610.17327/) (2026)

## Abstract (arXiv)

Role-playing large language models (LLMs) are expected to adopt a character's style while also respecting that character's knowledge boundaries. Prior evaluations detect character hallucination but rarely distinguish whether errors arise from failure to recognize a boundary or from failure to comply despite recognition. We introduce CHARM, a multicultural benchmark of 40 real and fictional characters drawn from five cultural-linguistic regions, and validated by native reviewers. It probes two boundary types, Temporal (historical vs. modern) and Cross-Universe (entities outside a character's narrative or historical universe), using abstention-enabled multiple-choice questions. We propose a two-stage evaluation that separates Boundary-Awareness (explicit recognition that a query is out of scope) from Boundary-Compliance (abstention when answering concrete questions). Evaluations across six LLMs show that hallucination is driven predominantly by compliance failures. Models frequently acknowledge that a query lies outside the character's knowledge yet still provide factual, out-of-character answers. By re-posing the same questions to the target character, we confirm that a large fraction of these cases are verified parametric overrides; the model stores the relevant fact but fails to suppress it. We also observe systematic cultural variation in these failures, consistent with imbalances in how characters from different regions are represented in model knowledge.