# Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects arXiv: https://arxiv.org/abs/2509.04794 ยท HTML: https://mlsanity.com/paper/2509.04794/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 25 of its 32 references that are in the map. - First posted on arXiv 5 Sep 2025. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 25 of 25) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020) - [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023) - [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023) - [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023) - [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021) - [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025) - [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023) - [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023) - [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023) - [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024) - [Can ChatGPT Assess Human Personalities? A General Evaluation Framework](https://mlsanity.com/paper/2303.01248/) (2023) - [The Better Angels of Machine Personality: How Personality Relates to LLM Safety](https://mlsanity.com/paper/2407.12344/) (2024) - [Exploring the Impact of Personality Traits on LLM Bias and Toxicity](https://mlsanity.com/paper/2502.12566/) (2025) - [Eliciting Personality Traits in Large Language Models](https://mlsanity.com/paper/2402.08341/) (2024) - [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024) - [UPLex: Fine-Grained Personality Control in Large Language Models via Unsupervised Lexical Modulation](https://mlsanity.com/paper/2310.16582/) (2023) - [P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts](https://mlsanity.com/paper/2406.12548/) (2024) - [From Text to Emoji: How PEFT-Driven Personality Manipulation Unleashes the Emoji Potential in LLMs](https://mlsanity.com/paper/2409.10245/) (2024) ## Built on by (most cited first, 4 of 4) - [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026) - [Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events](https://mlsanity.com/paper/2608.06485/) (2026) - [The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs](https://mlsanity.com/paper/2605.27382/) (2026) - [ICL Optimized Fragility](https://mlsanity.com/paper/2510.00300/) (2025) ## Abstract (arXiv) Personality manipulation in large language models (LLMs) is increasingly applied in customer service and agentic scenarios, yet its mechanisms and trade-offs remain unclear. We present a systematic study of personality control using the Big Five traits, comparing in-context learning (ICL), parameter-efficient fine-tuning (PEFT), and mechanistic steering (MS). Our contributions are fourfold. First, we construct a contrastive dataset with balanced high/low trait responses, enabling effective steering vector computation and fair cross-method evaluation. Second, we introduce a unified evaluation framework based on within-run Delta analysis that disentangles, reasoning capability, agent performance, and demographic bias across MMLU, GAIA, and BBQ benchmarks. Third, we develop trait purification techniques to separate openness from conscientiousness, addressing representational overlap in trait encoding. Fourth, we propose a three-level stability framework that quantifies method-, trait-, and combination-level robustness, offering practical guidance under deployment constraints. Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs: ICL achieves strong alignment with minimal capability loss, PEFT delivers the highest alignment at the cost of degraded task performance, and MS provides lightweight runtime control with competitive effectiveness. Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL, and personality encoding consolidating around intermediate layers. Taken together, these results establish personality manipulation as a multi-level probe into behavioral representation, linking surface conditioning, parameter encoding, and activation-level steering, and positioning mechanistic steering as a lightweight alternative to fine-tuning for both deployment and interpretability.