# Evaluating Alignment of Behavioral Dispositions in LLMs

arXiv: https://arxiv.org/abs/2602.11328 · HTML: https://mlsanity.com/paper/2602.11328/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 28 of 28)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models](https://mlsanity.com/paper/2601.10387/) (2026)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [Self-Assessment Tests are Unreliable Measures of LLM Personality](https://mlsanity.com/paper/2309.08163/) (2023)
- [Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench](https://mlsanity.com/paper/2310.01386/) (2023)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)
- [Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots](https://mlsanity.com/paper/2412.00207/) (2024)
- [ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models](https://mlsanity.com/paper/2406.04214/) (2024)
- [Reasoning Models are Test Exploiters: Rethinking Multiple-Choice](https://mlsanity.com/paper/2507.15337/) (2025)
- [An Invariant Learning Characterization of Controlled Text Generation](https://mlsanity.com/paper/2306.00198/) (2023)
- [Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models](https://mlsanity.com/paper/2509.16332/) (2025)

## Built on by (most cited first, 7 of 7)

- [Measuring Progress Toward AGI: A Cognitive Framework](https://mlsanity.com/paper/2605.28405/) (2026)
- [Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes](https://mlsanity.com/paper/2606.20708/) (2026)
- [Efficient Safety Alignment of Language Models via Latent Personality Traits](https://mlsanity.com/paper/2607.07918/) (2026)
- [Verifiable Social Reasoning for LLM Assistants](https://mlsanity.com/paper/2609.17496/) (2026)
- [NodeSynth: Socially Aligned Synthetic Data for AI Evaluation](https://mlsanity.com/paper/2605.14381/) (2026)
- [LLM Scheming Inversely Scales with Pretraining Language Coverage](https://mlsanity.com/paper/2607.24769/) (2026)
- [Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments](https://mlsanity.com/paper/2610.15634/) (2026)

## Abstract (arXiv)

As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dispositions: the underlying tendencies that shape responses in social contexts. We introduce STAR, a framework for studying how closely the dispositions expressed by LLMs align with those of humans. STAR builds on established psychological questionnaires, adapting their items into realistic advice-seeking scenarios, as self-report may not transfer to actual advisory behavior. Using STAR, we construct a dataset of 23k scenarios, each validated by 3 raters and annotated with preferences from 10 participants. Across 25 LLMs, we find that (1) when human consensus is high, frontier models can fail to reflect it in 15-20% of cases, and smaller models fail at substantially higher rates; (2) when humans disagree, LLM recommendations are substantially less diverse than human choices, both within individual models and even across models from different providers, potentially narrowing the range of options users are guided toward; (3) LLMs' self-reported values are poor predictors of their recommendations. To support future research we make our dataset and code publicly available.