# Stress-Testing Model Specs Reveals Character Differences among Language Models

arXiv: https://arxiv.org/abs/2510.07686 · HTML: https://mlsanity.com/paper/2510.07686/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions](https://mlsanity.com/paper/2504.15236/) (2025)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Inverse Constitutional AI: Compressing Preferences into Principles](https://mlsanity.com/paper/2406.06560/) (2024)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas](https://mlsanity.com/paper/2505.14633/) (2025)
- [C3AI: Crafting and Evaluating Constitutions for Constitutional AI](https://mlsanity.com/paper/2502.15861/) (2025)
- [SpecEval: Evaluating Model Adherence to Behavior Specifications](https://mlsanity.com/paper/2509.02464/) (2025)
- [Statutory Construction and Interpretation for Artificial Intelligence](https://mlsanity.com/paper/2509.01186/) (2025)

## Built on by (most cited first, 13 of 13)

- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [When Do LLM Preferences Predict Downstream Behavior?](https://mlsanity.com/paper/2602.18971/) (2026)
- [PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges](https://mlsanity.com/paper/2605.30803/) (2026)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't](https://mlsanity.com/paper/2605.01771/) (2026)
- [Probing Persona-Dependent Preferences in Language Models](https://mlsanity.com/paper/2605.13339/) (2026)
- [Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values](https://mlsanity.com/paper/2605.10365/) (2026)
- [Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies](https://mlsanity.com/paper/2604.15607/) (2026)
- [SCRuB: Social Concept Reasoning under Rubric-Based Evaluation](https://mlsanity.com/paper/2605.06444/) (2026)
- [Synthetic Persona Pretraining: Alignment from Token Zero](https://mlsanity.com/paper/2608.13482/) (2026)
- [BACH-V: Bridging Abstract and Concrete Human-Values in Large Language Models](https://mlsanity.com/paper/2601.14007/) (2026)
- [Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability](https://mlsanity.com/paper/2610.01168/) (2026)
- [Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning](https://mlsanity.com/paper/2610.01847/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly trained from AI constitutions and model specifications that establish behavioral guidelines and ethical principles. However, these specifications face critical challenges, including internal conflicts between principles and insufficient coverage of nuanced scenarios. We present a systematic methodology for stress-testing model character specifications, automatically identifying numerous cases of principle contradictions and interpretive ambiguities in current model specs. We stress test current model specs by generating scenarios that force explicit tradeoffs between competing value-based principles. Using a comprehensive taxonomy we generate diverse value tradeoff scenarios where models must choose between pairs of legitimate principles that cannot be simultaneously satisfied. We evaluate responses from twelve frontier LLMs across major providers (Anthropic, OpenAI, Google, xAI) and measure behavioral disagreement through value classification scores. Among these scenarios, we identify over 70,000 cases exhibiting significant behavioral divergence. Empirically, we show this high divergence in model behavior strongly predicts underlying problems in model specifications. Through qualitative analysis, we provide numerous example issues in current model specs such as direct contradiction and interpretive ambiguities of several principles. Additionally, our generated dataset also reveals both clear misalignment cases and false-positive refusals across all of the frontier models we study. Lastly, we also provide value prioritization patterns and differences of these models.