# Beyond Self-Reports: Multi-Observer Agents for Personality Assessment in Large Language Models

arXiv: https://arxiv.org/abs/2504.08399 · HTML: https://mlsanity.com/paper/2504.08399/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 44 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 6 of 6)

- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [Self-Assessment Tests are Unreliable Measures of LLM Personality](https://mlsanity.com/paper/2309.08163/) (2023)
- [LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices](https://mlsanity.com/paper/2403.10779/) (2024)
- [The Internet of People: A human and data-centric paradigm for the Next Generation Internet](https://mlsanity.com/paper/2205.13970/) (2022)

## Built on by (most cited first, 3 of 3)

- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Personality Expression Across Contexts: Linguistic and Behavioral Variation in LLM Agents](https://mlsanity.com/paper/2602.01063/) (2026)
- [On the Behavioral Traits of LLM Agents](https://mlsanity.com/paper/2609.32776/) (2026)

## Abstract (arXiv)

Self-report questionnaires have long been used to assess LLM personality traits, yet they fail to capture behavioral nuances due to biases and meta-knowledge contamination. This paper proposes a novel multi-observer framework for personality trait assessments in LLM agents that draws on informant-report methods in psychology. Instead of relying on self-assessments, we employ multiple observer agents. Each observer is configured with a specific relational context (e.g., family member, friend, or coworker) and engages the subject LLM in dialogue before evaluating its behavior across the Big Five dimensions. We show that these observer-report ratings align more closely with human judgments than traditional self-reports and reveal systematic biases in LLM self-assessments. We also found that aggregating responses from 5 to 7 observers reduces systematic biases and achieves optimal reliability. Our results highlight the role of relationship context in perceiving personality and demonstrate that a multi-observer paradigm offers a more reliable, context-sensitive approach to evaluating LLM personality traits.