# RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity

arXiv: https://arxiv.org/abs/2509.25897 · HTML: https://mlsanity.com/paper/2509.25897/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 30 of 45)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences](https://mlsanity.com/paper/2012.15738/) (2020)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation](https://mlsanity.com/paper/2502.09838/) (2025)
- [Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes](https://mlsanity.com/paper/2008.09094/) (2020)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [MoralBench: Moral Evaluation of LLMs](https://mlsanity.com/paper/2406.04428/) (2024)
- [Evaluating Cultural and Social Awareness of LLM Web Agents](https://mlsanity.com/paper/2410.23252/) (2024)
- [ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations](https://mlsanity.com/paper/2212.10409/) (2022)
- [Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?](https://mlsanity.com/paper/2308.15399/) (2023)
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](https://mlsanity.com/paper/2402.13605/) (2024)

## Built on by (most cited first, 4 of 4)

- [Social Dynamics as Critical Vulnerabilities that Undermine Objective Decision-Making in LLM Collectives](https://mlsanity.com/paper/2604.06091/) (2026)
- [Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values](https://mlsanity.com/paper/2605.10365/) (2026)
- [Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation](https://mlsanity.com/paper/2604.17020/) (2026)
- [Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignment](https://mlsanity.com/paper/2605.28188/) (2026)

## Abstract (arXiv)

People often encounter role conflicts -- social dilemmas where the expectations of multiple roles clash and cannot be simultaneously fulfilled. As large language models (LLMs) increasingly navigate these social dynamics, a critical research question emerges. When faced with such dilemmas, do LLMs prioritize dynamic contextual cues or the learned preferences? To address this, we introduce RoleConflictBench, a novel benchmark designed to measure the contextual sensitivity of LLMs in role conflict scenarios. To enable objective evaluation within this subjective domain, we employ situational urgency as a constraint for decision-making. We construct the dataset through a three-stage pipeline that generates over 13,000 realistic scenarios across 65 roles in five social domains by systematically varying the urgency of competing situations. This controlled setup enables us to quantitatively measure contextual sensitivity, determining whether model decisions align with the situational contexts or are overridden by the learned role preferences. Our analysis of 10 LLMs reveals that models substantially deviate from this objective baseline. Instead of responding to dynamic contextual cues, their decisions are predominantly governed by the preferences toward specific social roles.