# Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters arXiv: https://arxiv.org/abs/2211.06869 ยท HTML: https://mlsanity.com/paper/2211.06869/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 17 of its 32 references that are in the map. - First posted on arXiv 13 Nov 2022. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 17 of 17) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022) - [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018) - [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018) - [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016) - [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019) - [Personalized Dialogue Generation with Diversified Traits](https://mlsanity.com/paper/1901.09672/) (2019) - [A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data](https://mlsanity.com/paper/1911.04700/) (2019) - [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021) - [Towards Data Distillation for End-to-end Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.08923/) (2020) - [Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge](https://mlsanity.com/paper/2112.08619/) (2021) - [Self-supervised Dialogue Learning for Spoken Conversational Question Answering](https://mlsanity.com/paper/2106.02182/) (2021) - [Pchatbot: A Large-Scale Dataset for Personalized Chatbot](https://mlsanity.com/paper/2009.13284/) (2020) - [Profile Consistency Identification for Open-domain Dialogue Agents](https://mlsanity.com/paper/2009.09680/) (2020) - [COSPLAY: Concept Set Guided Personalized Dialogue Generation Across Both Party Personas](https://mlsanity.com/paper/2205.00872/) (2022) - [Neural Personalized Response Generation as Domain Adaptation](https://mlsanity.com/paper/1701.02073/) (2017) ## Built on by (most cited first, 16 of 16) - [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024) - [PersonaGym: Evaluating Persona Agents and LLMs](https://mlsanity.com/paper/2407.18416/) (2024) - [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023) - [BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data](https://mlsanity.com/paper/2410.16491/) (2024) - [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023) - [Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning](https://mlsanity.com/paper/2503.17662/) (2025) - [Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrieval](https://mlsanity.com/paper/2410.23041/) (2024) - [Mitigating Hallucination in Fictional Character Role-Play](https://mlsanity.com/paper/2406.17260/) (2024) - [From Good to Great: Improving Math Reasoning with Tool-Augmented Interleaf Prompting](https://mlsanity.com/paper/2401.05384/) (2023) - [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025) - [Characteristic AI Agents via Large Language Models](https://mlsanity.com/paper/2403.12368/) (2024) - [ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing](https://mlsanity.com/paper/2604.11103/) (2026) - [PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models](https://mlsanity.com/paper/2605.17044/) (2026) - [Learning a Single Token to Replace Long System Prompts in LLMs](https://mlsanity.com/paper/2511.23271/) (2025) - [Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension](https://mlsanity.com/paper/2302.13619/) (2023) - [TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation](https://mlsanity.com/paper/2608.11236/) (2026) ## Abstract (arXiv) In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.