# Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfare

arXiv: https://arxiv.org/abs/2509.07961 · HTML: https://mlsanity.com/paper/2509.07961/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 20 of 20)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [Taking AI Welfare Seriously](https://mlsanity.com/paper/2411.00986/) (2024)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Machine Psychology](https://mlsanity.com/paper/2303.13988/) (2023)
- [Towards Evaluating AI Systems for Moral Status Using Self-Reports](https://mlsanity.com/paper/2311.08576/) (2023)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Can LLMs make trade-offs involving stipulated pain and pleasure states?](https://mlsanity.com/paper/2411.02432/) (2024)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- ["Understanding AI": Semantic Grounding in Large Language Models](https://mlsanity.com/paper/2402.10992/) (2024)
- [On the Parallels Between Evolutionary Theory and the State of AI](https://mlsanity.com/paper/2505.23774/) (2025)

## Built on by (most cited first, 7 of 7)

- [When Do LLM Preferences Predict Downstream Behavior?](https://mlsanity.com/paper/2602.18971/) (2026)
- [No Reliable Evidence of Self-Reported Sentience in Small Large Language Models](https://mlsanity.com/paper/2601.15334/) (2026)
- [Beyond Mimicry: Preference Coherence in LLMs](https://mlsanity.com/paper/2511.13630/) (2025)
- [Incoherent Values? Probing LLM Preferences Through Parametric Variation](https://mlsanity.com/paper/2606.21102/) (2026)
- [The Pain Axis: LLMs Represent Self-Directed Harm and Act on It](https://mlsanity.com/paper/2609.16247/) (2026)
- [How much of a measured AI preference is the model, and how much is the instrument?](https://mlsanity.com/paper/2608.23641/) (2026)
- [Language Models Act on Hidden Valence](https://mlsanity.com/paper/2609.35591/) (2026)

## Abstract (arXiv)

We develop new experimental paradigms for measuring welfare in language models. We compare verbal reports of models about their preferences with preferences expressed through behavior when navigating a virtual environment and selecting conversation topics. We also test how costs and rewards affect behavior and whether responses to an eudaimonic welfare scale - measuring states such as autonomy and purpose in life - are stable across semantically equivalent prompts. Overall, we observed a notable degree of mutual support between our measures. The reliable correlations observed between stated preferences and behavior across conditions suggest that preference satisfaction can, in principle, serve as an empirically measurable welfare proxy in some of today's AI systems. Furthermore, our design offered an illuminating setting for qualitative observation of model behavior. Yet, the consistency between measures was more pronounced in some models and conditions than others and responses were changed by perturbations. Due to this, and the background uncertainty about the nature of welfare and the cognitive states (and welfare subjecthood) of language models, we are currently uncertain whether our methods successfully measure the welfare state of language models. Nevertheless, these findings highlight the feasibility of welfare measurement in language models, inviting further exploration.