# Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles

arXiv: https://arxiv.org/abs/2507.22168 · HTML: https://mlsanity.com/paper/2507.22168/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering](https://mlsanity.com/paper/2406.12334/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [PERSONA: A Reproducible Testbed for Pluralistic Alignment](https://mlsanity.com/paper/2407.17387/) (2024)
- [Do LLMs write like humans? Variation in grammatical and rhetorical styles](https://mlsanity.com/paper/2410.16107/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity](https://mlsanity.com/paper/2411.04427/) (2024)
- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [Evaluating the Diversity and Quality of LLM Generated Content](https://mlsanity.com/paper/2504.12522/) (2025)
- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users](https://mlsanity.com/paper/2406.17737/) (2024)
- [The Sociolinguistic Foundations of Language Modeling](https://mlsanity.com/paper/2407.09241/) (2024)
- [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025)

## Built on by (most cited first, 16 of 16)

- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026)
- [One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization](https://mlsanity.com/paper/2601.18572/) (2026)
- [Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM](https://mlsanity.com/paper/2603.18507/) (2026)
- [Prompt Fairness: Sub-group Disparities in LLMs](https://mlsanity.com/paper/2511.19956/) (2025)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks](https://mlsanity.com/paper/2603.21404/) (2026)
- [Do Ethical AI Principles Matter to Users? A Large-Scale Analysis of User Sentiment and Satisfaction](https://mlsanity.com/paper/2508.05913/) (2025)
- [CreAgentive: An Agent Workflow Driven Multi-Category Creative Generation Engine](https://mlsanity.com/paper/2509.26461/) (2025)
- [Voice Under Revision: Large Language Models and the Normalization of Personal Narrative](https://mlsanity.com/paper/2604.22142/) (2026)
- [Building Persona-Based Agents On Demand: Tailoring Multi-Agent Workflows to User Needs](https://mlsanity.com/paper/2604.27882/) (2026)
- [PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI](https://mlsanity.com/paper/2605.05682/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)
- [RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests](https://mlsanity.com/paper/2608.27831/) (2026)
- [AwarenessBench: Assessing Cognitive Capabilities of Language Models](https://mlsanity.com/paper/2609.35409/) (2026)
- [Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs](https://mlsanity.com/paper/2610.01428/) (2026)

## Abstract (arXiv)

Current benchmarks for evaluating Large Language Models (LLMs) often do not exhibit enough writing style diversity, with many adhering primarily to standardized conventions. Such benchmarks do not fully capture the rich variety of communication patterns exhibited by humans. Thus, it is possible that LLMs, which are optimized on these benchmarks, may demonstrate brittle performance when faced with "non-standard" input. In this work, we test this hypothesis by rewriting evaluation prompts using persona-based LLM prompting, a low-cost method to emulate diverse writing styles. Our results show that, even with identical semantic content, variations in writing style and prompt formatting significantly impact the estimated performance of the LLM under evaluation. Notably, we identify distinct writing styles that consistently trigger either low or high performance across a range of models and tasks, irrespective of model family, size, and recency. Our work offers a scalable approach to augment existing benchmarks, improving the external validity of the assessments they provide for measuring LLM performance across linguistic variations.