# Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios

arXiv: https://arxiv.org/abs/2503.01532 · HTML: https://mlsanity.com/paper/2503.01532/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2025.

## Built on by (most cited first, 26 of 26)

- [Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD](https://mlsanity.com/paper/2508.17450/) (2025)
- [Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions](https://mlsanity.com/paper/2512.12775/) (2025)
- [From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions](https://mlsanity.com/paper/2511.11789/) (2025)
- [Bias Association Discovery Framework for Open-Ended LLM Generations](https://mlsanity.com/paper/2508.01412/) (2025)
- [PakBBQ: A Culturally Adapted Bias Benchmark for QA](https://mlsanity.com/paper/2508.10186/) (2025)
- [BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models](https://mlsanity.com/paper/2510.11178/) (2025)
- [MOSAIC: Unveiling the Moral, Social and Individual Dimensions of Large Language Models](https://mlsanity.com/paper/2603.00048/) (2026)
- [Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation](https://mlsanity.com/paper/2604.17079/) (2026)
- [Beyond Keywords: Evaluating Large Language Model Classification of Nuanced Ableism](https://mlsanity.com/paper/2505.20500/) (2025)
- [MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs for Theory of Mind](https://mlsanity.com/paper/2506.14161/) (2025)
- [EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Norms](https://mlsanity.com/paper/2507.20264/) (2025)
- [Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents](https://mlsanity.com/paper/2508.00742/) (2025)
- [Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues](https://mlsanity.com/paper/2601.12868/) (2026)
- [Fine-Grained Interpretation of Political Opinions in Large Language Models](https://mlsanity.com/paper/2506.04774/) (2025)
- [Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screening](https://mlsanity.com/paper/2507.11548/) (2025)
- [Demographic Biases and Gaps in the Perception of Sexism in Large Language Models](https://mlsanity.com/paper/2508.18245/) (2025)
- [Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset](https://mlsanity.com/paper/2510.01219/) (2025)
- [CHOIR: Collaborative Harmonization fOr Inference Robustness](https://mlsanity.com/paper/2510.22475/) (2025)
- [Large language models for spreading dynamics in complex systems](https://mlsanity.com/paper/2602.08085/) (2026)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues](https://mlsanity.com/paper/2604.01925/) (2026)
- [Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups](https://mlsanity.com/paper/2605.08671/) (2026)
- [Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?](https://mlsanity.com/paper/2605.17694/) (2026)
- [Pareto-Guided Teacher Alignment for Fair Personalized Text Generation](https://mlsanity.com/paper/2606.10126/) (2026)
- [Beyond Mode Collapse: Generating Diverse Synthetic Expert Conversations via Generative Flow Networks](https://mlsanity.com/paper/2609.38359/) (2026)
- [Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulation](https://mlsanity.com/paper/2610.04676/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in simulating human behaviour and social intelligence. However, they risk perpetuating societal biases, especially when demographic information is involved. We introduce a novel framework using cosine distance to measure semantic shifts in responses and an LLM-judged Preference Win Rate (WR) to assess how demographic prompts affect response quality across power-disparate social scenarios. Evaluating five LLMs over 100 diverse social scenarios and nine demographic axes, our findings suggest a "default persona" bias toward middle-aged, able-bodied, native-born, Caucasian, atheistic males with centrist views. Moreover, interactions involving specific demographics are associated with lower-quality responses. Lastly, the presence of power disparities increases variability in response semantics and quality across demographic groups, suggesting that implicit biases may be heightened under power-imbalanced conditions. These insights expose the demographic biases inherent in LLMs and offer potential paths toward future bias mitigation efforts in LLMs.