Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing
Not on the map. Why?
Abstract (arXiv)
Role-Playing Agents (RPAs) have shown remarkable performance in various applications, yet they often struggle to recognize and appropriately respond to hard queries that conflict with their role-play knowledge. To investigate RPAs' performance when faced with different types of conflicting requests, we develop an evaluation benchmark that includes contextual knowledge conflicting requests, parametric knowledge conflicting requests, and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer appropriately without over-refusing. Through extensive evaluation, we find that most RPAs behave significant performance gaps toward different conflict requests. To elucidate the reasons, we conduct an in-depth representation-level analysis of RPAs under various conflict scenarios. Our findings reveal the existence of rejection regions and direct response regions within the model's forwarding representation, and thus influence the RPA's final response behavior. Therefore, we introduce a lightweight representation editing approach that conveniently shifts conflicting requests to the rejection region, thereby enhancing the model's refusal accuracy. The experimental results validate the effectiveness of our editing method, improving RPAs' refusal ability of conflicting requests while maintaining their general role-playing capabilities.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 25 Sep 2024.
Builds on
45 of 52 references.
References · 45
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Mistral 7Bcited by 3,065
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2024Qwen2 Technical Reportcited by 2,076
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2023The Internal State of an LLM Knows When It's Lyingcited by 644
- 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
- 2024LoRA Learns Less and Forgets Lesscited by 313
- 2024Knowledge Conflicts for LLMs: A Surveycited by 274
- 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
33 more
- 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
- 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
- 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
- 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 139
- 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
- 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
- 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
- 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
- 2019Personalized Dialogue Generation with Diversified Traitscited by 73
- 2023Alignment for Honestycited by 72
- 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
- 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
- 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
- 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 58
- 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
- 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
- 2024Can AI Assistants Know What They Don't Know?cited by 47
- 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
- 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
- 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
- 2023CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Modelscited by 34
- 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
- 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
- 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
- 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanationscited by 30
- 2020Pchatbot: A Large-Scale Dataset for Personalized Chatbotcited by 26
- 2020Profile Consistency Identification for Open-domain Dialogue Agentscited by 22
- 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 21
- 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
- 2024Mitigating Hallucination in Fictional Character Role-Playcited by 17
- 2023NarrativePlay: Interactive Narrative Understandingcited by 10
- 2024Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agentscited by 7
- 2024Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvementcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.