# Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?

arXiv: https://arxiv.org/abs/2506.00751 · HTML: https://mlsanity.com/paper/2506.00751/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 31 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [Can large language models explore in-context?](https://mlsanity.com/paper/2403.15371/) (2024)
- [Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection](https://mlsanity.com/paper/2308.10819/) (2023)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processes](https://mlsanity.com/paper/2404.17218/) (2024)
- [A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions](https://mlsanity.com/paper/2409.16430/) (2024)
- [Understanding Bias in Large-Scale Visual Datasets](https://mlsanity.com/paper/2412.01876/) (2024)
- [Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing](https://mlsanity.com/paper/2504.01282/) (2025)
- [Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Training](https://mlsanity.com/paper/2412.02857/) (2024)
- [The Reliability of LLMs for Medical Diagnosis: An Examination of Consistency, Manipulation, and Contextual Awareness](https://mlsanity.com/paper/2503.10647/) (2025)

## Built on by (most cited first, 10 of 10)

- [Asymmetric Goal Drift in Coding Agents Under Value Conflict](https://mlsanity.com/paper/2603.03456/) (2026)
- [Probing Persona-Dependent Preferences in Language Models](https://mlsanity.com/paper/2605.13339/) (2026)
- [Pressure Reveals Character: Behavioural Alignment Evaluation at Depth](https://mlsanity.com/paper/2602.20813/) (2026)
- [When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models](https://mlsanity.com/paper/2606.22974/) (2026)
- [What LLM Agents Say When No One Is Watching: Social Structure and Latent Objective Emergence in Multi-Agent Debates](https://mlsanity.com/paper/2607.02507/) (2026)
- [Superficial Beliefs in LLM Decision-Making](https://mlsanity.com/paper/2606.11016/) (2026)
- [Strangers to Themselves: What Language Models Say About Themselves Is Generic](https://mlsanity.com/paper/2609.09899/) (2026)
- [AgentPersonaBench: Benchmarking Persona-Driven User Simulation](https://mlsanity.com/paper/2610.04379/) (2026)
- [Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment](https://mlsanity.com/paper/2610.08670/) (2026)
- [Beyond the greenwash: do LLMs' ecologically sound decisions hold up under pressure?](https://mlsanity.com/paper/2610.16043/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) highlight the need to align their behaviors with human values. A critical, yet understudied, issue is the potential divergence between an LLM's stated preferences (its reported alignment with general principles) and its revealed preferences (inferred from decisions in contextualized scenarios). Such deviations raise fundamental concerns for the interpretability, trustworthiness, reasoning transparency, and ethical deployment of LLMs, particularly in high-stakes applications. This work formally defines and proposes a method to measure this preference deviation. We investigate how LLMs may activate different guiding principles in specific contexts, leading to choices that diverge from previously stated general principles. Our approach involves crafting a rich dataset of well-designed prompts as a series of forced binary choices and presenting them to LLMs. We compare LLM responses to general principle prompts stated preference with LLM responses to contextualized prompts revealed preference, using metrics like KL divergence to quantify the deviation. We repeat the analysis across different categories of preferences and on four mainstream LLMs and find that a minor change in prompt format can often pivot the preferred choice regardless of the preference categories and LLMs in the test. This prevalent phenomenon highlights the lack of understanding and control of the LLM decision-making competence. Our study will be crucial for integrating LLMs into services, especially those that interact directly with humans, where morality, fairness, and social responsibilities are crucial dimensions. Furthermore, identifying or being aware of such deviation will be critically important as LLMs are increasingly envisioned for autonomous agentic tasks where continuous human evaluation of all LLMs' intermediary decision-making steps is impossible.