# Large Language Model Agent Personality and Response Appropriateness: Evaluation by Human Linguistic Experts, LLM-as-Judge, and Natural Language Processing Model

arXiv: https://arxiv.org/abs/2510.23875 · HTML: https://mlsanity.com/paper/2510.23875/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models](https://mlsanity.com/paper/2307.16180/) (2023)
- [PANDORA Talks: Personality and Demographics on Reddit](https://mlsanity.com/paper/2004.04460/) (2020)
- [Identifying and Manipulating the Personality Traits of Language Models](https://mlsanity.com/paper/2212.10276/) (2022)
- [Continuous Output Personality Detection Models via Mixed Strategy Training](https://mlsanity.com/paper/2406.16223/) (2024)

## Abstract (arXiv)

While Large Language Model (LLM)-based agents can be used to create highly engaging interactive applications through prompting personality traits and contextual data, effectively assessing their personalities has proven challenging. This novel interdisciplinary approach addresses this gap by combining agent development and linguistic analysis to assess the prompted personality of LLM-based agents in a poetry explanation task. We developed a novel, flexible question bank, informed by linguistic assessment criteria and human cognitive learning levels, offering a more comprehensive evaluation than current methods. By evaluating agent responses with natural language processing models, other LLMs, and human experts, our findings illustrate the limitations of purely deep learning solutions and emphasize the critical role of interdisciplinary design in agent development.