# Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models

arXiv: https://arxiv.org/abs/2502.07077 · HTML: https://mlsanity.com/paper/2502.07077/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles](https://mlsanity.com/paper/2407.00870/) (2024)
- [People cannot distinguish GPT-4 from a human in a Turing test](https://mlsanity.com/paper/2405.08007/) (2024)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions](https://mlsanity.com/paper/2409.16427/) (2024)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language Models](https://mlsanity.com/paper/2405.06079/) (2024)
- [Mirages: On Anthropomorphism in Dialogue Systems](https://mlsanity.com/paper/2305.09800/) (2023)
- [AnthroScore: A Computational Linguistic Measure of Anthropomorphism](https://mlsanity.com/paper/2402.02056/) (2024)
- [LearnLM: Improving Gemini for Learning](https://mlsanity.com/paper/2412.16429/) (2024)
- [Evaluating Generative AI Systems is a Social Science Measurement Challenge](https://mlsanity.com/paper/2411.10939/) (2024)
- [Characterizing and modeling harms from interactions with design patterns in AI interfaces](https://mlsanity.com/paper/2404.11370/) (2024)
- ["I Am the One and Only, Your Cyber BFF": Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI](https://mlsanity.com/paper/2410.08526/) (2024)

## Built on by (most cited first, 25 of 25)

- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Training language models to be warm and empathetic makes them less reliable and more sycophantic](https://mlsanity.com/paper/2507.21919/) (2025)
- [Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illness](https://mlsanity.com/paper/2507.19218/) (2025)
- [Thinking beyond the anthropomorphic paradigm benefits LLM research](https://mlsanity.com/paper/2502.09192/) (2025)
- [Generative Value Conflicts Reveal LLM Priorities](https://mlsanity.com/paper/2509.25369/) (2025)
- [Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design](https://mlsanity.com/paper/2508.17573/) (2025)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [INTIMA: A Benchmark for Human-AI Companionship Behavior](https://mlsanity.com/paper/2508.09998/) (2025)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Humanlike AI Design Increases Anthropomorphism but Yields Divergent Outcomes on Engagement and Trust Globally](https://mlsanity.com/paper/2512.17898/) (2025)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [Verbalizing LLMs' assumptions to explain and control sycophancy](https://mlsanity.com/paper/2604.03058/) (2026)
- [Artificial Intelligence in Deliberation: The AI Penalty and the Emergence of a New Deliberative Divide](https://mlsanity.com/paper/2503.07690/) (2025)
- [Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals](https://mlsanity.com/paper/2602.03061/) (2026)
- [Not Minds, but Signs: Reframing LLMs through Semiotics](https://mlsanity.com/paper/2505.17080/) (2025)
- [If LLMs Have Human-Like Attributes, Then So Does Age of Empires II](https://mlsanity.com/paper/2605.31514/) (2026)
- [Deflating Deflationism: A Critical Perspective on Debunking Arguments Against LLM Mentality](https://mlsanity.com/paper/2506.13403/) (2025)
- [BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation format](https://mlsanity.com/paper/2509.02655/) (2025)
- [ReIn: Conversational Error Recovery with Reasoning Inception](https://mlsanity.com/paper/2602.17022/) (2026)
- [EUDAIMONIA: Evaluating Undesirable Dynamics in AI](https://mlsanity.com/paper/2605.30654/) (2026)
- [How Value Induction Reshapes LLM Behaviour](https://mlsanity.com/paper/2605.07925/) (2026)
- [Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference](https://mlsanity.com/paper/2606.07897/) (2026)
- [Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts](https://mlsanity.com/paper/2606.18258/) (2026)
- [AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction](https://mlsanity.com/paper/2609.37853/) (2026)

## Abstract (arXiv)

The tendency of users to anthropomorphise large language models (LLMs) is of growing interest to AI developers, researchers, and policy-makers. Here, we present a novel method for empirically evaluating anthropomorphic LLM behaviours in realistic and varied settings. Going beyond single-turn static benchmarks, we contribute three methodological advances in state-of-the-art (SOTA) LLM evaluation. First, we develop a multi-turn evaluation of 14 anthropomorphic behaviours. Second, we present a scalable, automated approach by employing simulations of user interactions. Third, we conduct an interactive, large-scale human subject study (N=1101) to validate that the model behaviours we measure predict real users' anthropomorphic perceptions. We find that all SOTA LLMs evaluated exhibit similar behaviours, characterised by relationship-building (e.g., empathy and validation) and first-person pronoun use, and that the majority of behaviours only first occur after multiple turns. Our work lays an empirical foundation for investigating how design choices influence anthropomorphic model behaviours and for progressing the ethical debate on the desirability of these behaviours. It also showcases the necessity of multi-turn evaluations for complex social phenomena in human-AI interaction.