# Incoherent Values? Probing LLM Preferences Through Parametric Variation

arXiv: https://arxiv.org/abs/2606.21102 · HTML: https://mlsanity.com/paper/2606.21102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 32)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes](https://mlsanity.com/paper/2008.09094/) (2020)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Examining Identity Drift in Conversations of LLM Agents](https://mlsanity.com/paper/2412.00804/) (2024)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes](https://mlsanity.com/paper/2510.16380/) (2025)
- [Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundations](https://mlsanity.com/paper/2405.11100/) (2024)
- [Attributions toward Artificial Agents in a modified Moral Turing Test](https://mlsanity.com/paper/2406.11854/) (2024)
- [Consistency Training Helps Stop Sycophancy and Jailbreaks](https://mlsanity.com/paper/2510.27062/) (2025)
- [CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas](https://mlsanity.com/paper/2604.15267/) (2026)
- [Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs](https://mlsanity.com/paper/2506.13082/) (2025)
- [Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfare](https://mlsanity.com/paper/2509.07961/) (2025)
- [The Artificial Self: Characterising the landscape of AI identity](https://mlsanity.com/paper/2603.11353/) (2026)
- [Your Language Model Secretly Contains Personality Subnetworks](https://mlsanity.com/paper/2602.07164/) (2026)
- [Talk, Judge, Cooperate: Gossip-Driven Indirect Reciprocity in Self-Interested LLM Agents](https://mlsanity.com/paper/2602.07777/) (2026)

## Built on by (most cited first, 2 of 2)

- [Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems](https://mlsanity.com/paper/2608.05015/) (2026)
- [An Investigation of Model Coherence: Narrow Finetunes Contradict Themselves Under Resampling](https://mlsanity.com/paper/2610.12129/) (2026)

## Abstract (arXiv)

To trust another autonomous entity -- human or AI -- it helps to know that how it acts given one set of reasons is at least somewhat predictive of how it would act under others. It is hard to trust someone with incoherent values. Some think of Large Language Models as merely stochastic text generators with no evaluative core -- superpositions of billions of possible characters, not one stable identity. But others have argued that LLMs *do* have stable, emergent values, which can be elicited by presenting them with a series of forced choices between arbitrary statements, and which emerge as a function of model scale. In this paper, we test this thesis by presenting LLMs with parametric variations on those forced choices. We reason that if a model genuinely prefers A to B, then except in unusual circumstances it should also reject B in favor of an augmented version of A, which has more of what makes A desirable -- A++. Our results indicate that earlier attributions of coherence may have overstated their case. Even the most capable models exhibit significant incoherence, and coherence does not appear to emerge as a result of underlying model capability. We do, however, find that models given time to reason are less incoherent than those with thinking disabled. More generally, we develop a novel framework for eliciting and evaluating coherent values, which can be used both to assess how trustworthy current models are, and -- in future work -- to provide reward signal that can be used for making more coherent agents.