# RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

arXiv: https://arxiv.org/abs/2409.16727 · HTML: https://mlsanity.com/paper/2409.16727/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [CEM: Commonsense-aware Empathetic Response Generation](https://mlsanity.com/paper/2109.05739/) (2021)
- [Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models](https://mlsanity.com/paper/2304.13835/) (2023)
- [From Role-Play to Drama-Interaction: An LLM Solution](https://mlsanity.com/paper/2405.14231/) (2024)
- [IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generation](https://mlsanity.com/paper/2407.01093/) (2024)
- [DialogBench: Evaluating LLMs as Human-like Dialogue Systems](https://mlsanity.com/paper/2311.01677/) (2023)
- [Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Persona](https://mlsanity.com/paper/2305.11482/) (2023)
- [TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models](https://mlsanity.com/paper/2405.18027/) (2024)
- [Mitigating Hallucination in Fictional Character Role-Play](https://mlsanity.com/paper/2406.17260/) (2024)
- [Identity-Driven Hierarchical Role-Playing Agents](https://mlsanity.com/paper/2407.19412/) (2024)

## Built on by (most cited first, 21 of 21)

- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [An Embarrassingly Simple Defense Against LLM Abliteration Attacks](https://mlsanity.com/paper/2505.19056/) (2025)
- [Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025](https://mlsanity.com/paper/2506.12430/) (2025)
- [Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models](https://mlsanity.com/paper/2504.21038/) (2025)
- [Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is](https://mlsanity.com/paper/2507.21820/) (2025)
- [HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing](https://mlsanity.com/paper/2511.08017/) (2025)
- [Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills](https://mlsanity.com/paper/2604.25109/) (2026)
- [JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks](https://mlsanity.com/paper/2603.01291/) (2026)
- [StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation](https://mlsanity.com/paper/2601.04740/) (2026)
- [FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline](https://mlsanity.com/paper/2510.06800/) (2025)
- [Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense](https://mlsanity.com/paper/2601.03594/) (2026)
- [When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints](https://mlsanity.com/paper/2604.16916/) (2026)
- [GraphAttack: Exploiting Representational Blindspots in LLM Safety Mechanisms](https://mlsanity.com/paper/2504.13052/) (2025)
- [Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills](https://mlsanity.com/paper/2605.29354/) (2026)
- [Voices Across Registers: Corpus-Conditioned Vernacular Jailbreaks against Aligned LLMs via Fanfiction Subgenres](https://mlsanity.com/paper/2606.04483/) (2026)
- [DoubtProbe: Black-Box Jailbreak Defense via Structural Verification and Semantic Auditing](https://mlsanity.com/paper/2606.16527/) (2026)
- [CHARM: Character Hallucination for Multicultural Role Play Benchmark](https://mlsanity.com/paper/2609.01352/) (2026)
- [Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs](https://mlsanity.com/paper/2604.08846/) (2026)
- [One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent](https://mlsanity.com/paper/2605.05630/) (2026)
- [Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes](https://mlsanity.com/paper/2608.23640/) (2026)
- [Trust Me, I'm Your Developer: Self-Issued Authentication in Large Language Models](https://mlsanity.com/paper/2609.03247/) (2026)

## Abstract (arXiv)

Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.