When Do LLM Preferences Predict Downstream Behavior?
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 32 references that are in the map. First posted on arXiv 21 Feb 2026.
Builds on
24 of 32 references.
Closest ancestors · 1
Same direction, newest first
- 2025Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfare9 Sep
Other references · 23
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2025Why Language Models Hallucinatecited by 205
- 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
- 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
- 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
11 more
- 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
- 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
- 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
- 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
- 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 17
- 2025Lessons from a Chimp: AI "Scheming" and the Quest for Ape Languagecited by 16
- 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
- 2025Stress-Testing Model Specs Reveals Character Differences among Language Modelscited by 10
- 2025Safety Cases: A Scalable Approach to Frontier AI Safetycited by 8
- 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
- 2024Inertia in Moral and Value Judgments of Large Language Modelscited by 2
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 4
- 2026Can Revealed Preferences Clarify LLM Alignment and Steering?cited by 4
- 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Modelscited by 1
- 2026High-Stakes Decisions with Language Models: Insights from Emergency Triagecited by 0
- 2026Strangers to Themselves: What Language Models Say About Themselves Is Genericcited by 0
- 2026Strangers to Themselves: What Language Models Say About Themselves Is Generic9 Sep 2026
- 2026High-Stakes Decisions with Language Models: Insights from Emergency Triage2 Aug 2026
- 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models22 Jun 2026
- 2026Can Revealed Preferences Clarify LLM Alignment and Steering?8 May 2026
- 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious17 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Mimicry: Preference Coherence in LLMs
- 2025Risk Profiling and Modulation for LLMs
- 2023Do LLM Agents Exhibit Social Behavior?
- 2025From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLM
- 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
- 2026Misaligned by Reward: Socially Undesirable Preferences in LLMs
- 2026LLM-Derived Preference Judgments Are Not Self-Consistent
- 2025neuralFOMO: Can LLMs Handle Being Second Best? Measuring Envy-Like Preferences in Multi-Agent Settings