mlsanityData to 14 Oct 2026

Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

arXiv 2607.20270 ↗First posted 22 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 18 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

12 of 18 references.

References · 12

  1. 2020Aligning AI With Shared Human Valuescited by 723
  2. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  3. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  4. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  5. 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 51
  6. 2023The Touch'e23-ValueEval Dataset for Identifying Human Values behind Argumentscited by 28
  7. 2024ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Modelscited by 19
  8. 2025Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Itemscited by 18
  9. 2024Measuring Human and AI Values Based on Generative Psychometrics with Large Language Modelscited by 13
  10. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  11. 2021Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company's Reputationcited by 6
  12. 2026Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfercited by 2

Built on by

Not cited yet.