Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Not on the map. Why?
Abstract (arXiv)
Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose Psi-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in Psi-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on Psi-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 1 Jun 2026.
Builds on
34 of 49 references.
References · 34
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
- 2023LaMP: When Large Language Models Meet Personalizationcited by 437
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
- 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
- 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
- 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
- 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 92
- 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 88
- 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
- 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
- 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
22 more
- 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
- 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
- 2024Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Languagecited by 31
- 2020Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategiescited by 29
- 2025Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generationcited by 27
- 2025Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Modelscited by 27
- 2025A Survey of Personalization: From RAG to Agentcited by 27
- 2025Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMscited by 21
- 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 21
- 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Modelscited by 20
- 2025A Personalized Conversational Benchmark: Towards Simulating Personalized Conversationscited by 17
- 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
- 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 16
- 2025Personalize Your LLM: Fake it then Align itcited by 15
- 2025It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topicscited by 14
- 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
- 2023Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastinationcited by 12
- 2025ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mindcited by 12
- 2025Towards Strategic Persuasion with Language Modelscited by 10
- 2025Among Them: A game-based framework for assessing persuasion capabilities of LLMscited by 8
- 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 7
- 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation
- 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
- 2026Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions
- 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
- 2026Personality-Aware Reinforcement Learning for Persuasive Dialogue with LLM-Driven Simulation
- 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
- 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
- 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue