Large Language Model Agent Personality and Response Appropriateness: Evaluation by Human Linguistic Experts, LLM-as-Judge, and Natural Language Processing Model
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
While Large Language Model (LLM)-based agents can be used to create highly engaging interactive applications through prompting personality traits and contextual data, effectively assessing their personalities has proven challenging. This novel interdisciplinary approach addresses this gap by combining agent development and linguistic analysis to assess the prompted personality of LLM-based agents in a poetry explanation task. We developed a novel, flexible question bank, informed by linguistic assessment criteria and human cognitive learning levels, offering a more comprehensive evaluation than current methods. By evaluating agent responses with natural language processing models, other LLMs, and human experts, our findings illustrate the limitations of purely deep learning solutions and emphasize the critical role of interdisciplinary design in agent development.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 10 of its 35 references that are in the map. First posted on arXiv 27 Oct 2025.
Builds on
10 of 35 references.
Closest ancestors · 1
Same direction, newest first
Other references · 9
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
- 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
- 2023Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Modelscited by 56
- 2020PANDORA Talks: Personality and Demographics on Redditcited by 40
- 2022Identifying and Manipulating the Personality Traits of Language Modelscited by 40
- 2024Continuous Output Personality Detection Models via Mixed Strategy Trainingcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
- 2024Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models
- 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
- 2026Structured Personality Control and Adaptation for LLM Agents
- 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
- 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Review
- 2025The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents
- 2025Giving AI Personalities Leads to More Human-Like Reasoning