STONIC: A Layered Measurement Contract for LLM Value Profiling
Not on the map. Why?
Abstract (arXiv)
LLM value studies often merge questionnaire ratings, pairwise choices, and values inferred from generated text into one profile. That merge assumes that the three observations describe the same stable preference. STONIC tests this assumption on 5,144 situations from four banks and 35 fixed model configurations. It compares responses rated in isolation, choices made under counterbalanced conflict, spontaneous answers, and later choices between a model's own answer and authored alternatives. 10 of 17 configurations with usable behavioral data preserve the endorsement-choice relation across banks. Every one of the 17 eligible configurations prefers its own earlier answer (median effect 0.790), although option position changes the choice rate in every eligible configuration. Profile shape transfers most strongly from ratings to conflict choices and weakens for spontaneous text. Three-way annotation of 200 L3 responses provides a task-local check of the semantic audit: FULCRA agrees most closely with the human majority, while DeBERTa retains useful rank information after calibration. Hidden states encode the completed decision more clearly than the prompt alone. Thus the models show reproducible behavioral continuity, but the evidence does not support one scorer-independent value identity across interfaces.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
20 of 36 references.
References · 20
- 2018Datasheets for Datasetscited by 1,386
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2018Model Cards for Model Reportingcited by 1,080
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2020Aligning AI With Shared Human Valuescited by 723
- 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
- 2019Measurement and Fairnesscited by 308
- 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
- 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
- 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
- 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
- 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
8 more
- 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
- 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
- 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 51
- 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 50
- 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 46
- 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
- 2025Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Itemscited by 18
- 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
Built on by
Citing papers per year: 2026: 1.
1 paper.