# Generative Value Conflicts Reveal LLM Priorities

arXiv: https://arxiv.org/abs/2509.25369 · HTML: https://mlsanity.com/paper/2509.25369/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 68 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 42)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [In conversation with Artificial Intelligence: aligning language models with human values](https://mlsanity.com/paper/2209.00731/) (2022)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions](https://mlsanity.com/paper/2504.15236/) (2025)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions](https://mlsanity.com/paper/2409.16427/) (2024)
- [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025)

## Built on by (most cited first, 13 of 13)

- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [When Do LLM Preferences Predict Downstream Behavior?](https://mlsanity.com/paper/2602.18971/) (2026)
- [Can Revealed Preferences Clarify LLM Alignment and Steering?](https://mlsanity.com/paper/2605.08556/) (2026)
- [Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communities](https://mlsanity.com/paper/2604.05339/) (2026)
- [Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest](https://mlsanity.com/paper/2604.08525/) (2026)
- [High-Stakes Decisions with Language Models: Insights from Emergency Triage](https://mlsanity.com/paper/2608.01361/) (2026)
- [Visual Distraction Undermines Moral Reasoning in Vision-Language Models](https://mlsanity.com/paper/2603.16445/) (2026)
- [Coherence Maximization Improves Pluralistic Alignment](https://mlsanity.com/paper/2606.03110/) (2026)
- [A Geometric Perspective on Stabilizing Value Conflict Resolution](https://mlsanity.com/paper/2607.17946/) (2026)
- [Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture](https://mlsanity.com/paper/2608.14566/) (2026)
- [From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societies](https://mlsanity.com/paper/2609.22600/) (2026)
- [ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs](https://mlsanity.com/paper/2609.38260/) (2026)
- [Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning](https://mlsanity.com/paper/2610.01847/) (2026)

## Abstract (arXiv)

Past work seeks to align large language model (LLM)-based assistants with a target set of values, but such assistants are frequently forced to make tradeoffs between values when deployed. In response to the scarcity of value conflict in existing alignment datasets, we introduce ConflictScope, an automatic pipeline to evaluate how LLMs prioritize different values. Given a user-defined value set, ConflictScope automatically generates scenarios in which a language model faces a conflict between two values sampled from the set. It then prompts target models with an LLM-written "user prompt" and evaluates their free-text responses to elicit a ranking over values in the value set. Comparing results between multiple-choice and open-ended evaluations, we find that models shift away from supporting protective values, such as harmlessness, and toward supporting personal values, such as user autonomy, in more open-ended value conflict settings. However, including detailed value orderings in models' system prompts improves alignment with a target ranking by 14%, showing that system prompting can achieve moderate success at aligning LLM behavior under value conflict. Our work demonstrates the importance of evaluating value prioritization in models and provides a foundation for future work in this area.