mlsanityData to 14 Oct 2026

CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

arXiv 2504.10823 ↗First posted 15 Apr 2025cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Navigating dilemmas involving conflicting values is challenging even for humans in high-stakes domains, let alone for AI, yet prior work has been limited to everyday scenarios. To close this gap, we introduce CLASH (Character perspective-based LLM Assessments in Situations with High-stakes), a meticulously curated dataset consisting of 345 high-impact dilemmas along with 3,795 individual perspectives of diverse values. CLASH enables the study of critical yet underexplored aspects of value-based decision-making processes, including understanding of decision ambivalence and psychological discomfort as well as capturing the temporal shifts of values in the perspectives of characters. By benchmarking 14 non-thinking and thinking models, we uncover several key findings. (1) Even strong proprietary models, such as GPT-5 and Claude-4-Sonnet, struggle with ambivalent decisions, achieving only 24.06 and 51.01 accuracy. (2) Although LLMs reasonably predict psychological discomfort, they do not adequately comprehend perspectives involving value shifts. (3) Cognitive behaviors that are effective in the math-solving and game strategy domains do not transfer to value reasoning. Instead, new failure patterns emerge, including early commitment and overcommitment. (4) The steerability of LLMs towards a given value is significantly correlated with their value preferences. (5) Finally, LLMs exhibit greater steerability when reasoning from a third-party perspective, although certain values (e.g., safety) benefit uniquely from first-person framing.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 37 of its 81 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

37 of 81 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  7. 2023Whose Opinions Do Language Models Reflect?cited by 773
  8. 2020Aligning AI With Shared Human Valuescited by 723
  9. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  10. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  11. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  12. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 222
25 more
  1. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  2. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  3. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  4. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  5. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  6. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  7. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  8. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  9. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  10. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  11. 2023Word Embeddings Are Steers for Language Modelscited by 70
  12. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
  13. 2023On the steerability of large language models toward data-driven personascited by 53
  14. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  15. 2023A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3cited by 46
  16. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  17. 2024Moral Alignment for LLM Agentscited by 39
  18. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  19. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 22
  20. 2025AI Alignment at Your Discretioncited by 20
  21. 2024Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domaincited by 14
  22. 2025Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoralcited by 14
  23. 2024GreedLlama: Performance of Financial Value-Aligned Large Language Models in Moral Reasoningcited by 9
  24. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
  25. 2024Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignmentcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  2. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  3. 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
  4. 2025RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivitycited by 4
  5. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 4
  6. 2026Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Valuescited by 2
  7. 2025VIVA+: Human-Centered Situational Decision-Makingcited by 0
  8. 2026MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoningcited by 0
  9. 2026Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversationcited by 0
  10. 2026FairLMs: A Turnkey Library for Fairness in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
  2. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  3. 2025Humans expect rationality and cooperation from LLM opponents in strategic games
  4. 2026Probing Outcome-Level Resemblance and Mechanism-Level Alignment in LLM Risk Decisions: Evidence from the St. Petersburg Game
  5. 2026Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
  6. 2025Beyond Nash Equilibrium: Bounded Rationality of LLMs and humans in Strategic Decision-making
  7. 2024Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models
  8. 2024Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games