# Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questions

arXiv: https://arxiv.org/abs/2508.11414 · HTML: https://mlsanity.com/paper/2508.11414/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 10 of 10)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items](https://mlsanity.com/paper/2505.01015/) (2025)
- [Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning](https://mlsanity.com/paper/2503.05212/) (2025)
- [Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework](https://mlsanity.com/paper/2506.14948/) (2025)
- [Understanding How Value Neurons Shape the Generation of Specified Values in LLMs](https://mlsanity.com/paper/2505.17712/) (2025)
- [Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification](https://mlsanity.com/paper/2410.05559/) (2024)
- [NeurIPS 2023 LLM Efficiency Fine-tuning Competition](https://mlsanity.com/paper/2503.13507/) (2025)
- [Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations](https://mlsanity.com/paper/2505.19674/) (2025)

## Abstract (arXiv)

Large language models implicitly encode preferences over human values, yet steering them often requires large training data. In this work, we investigate a simple approach: Can we reliably modify a model's value system in downstream behavior by training it to answer value survey questions accordingly? We first construct value profiles of several open-source LLMs by asking them to rate a series of value-related descriptions spanning 20 distinct human values, which we use as a baseline for subsequent experiments. We then investigate whether the value system of a model can be governed by fine-tuning on the value surveys. We evaluate the effect of finetuning on the model's behavior in two ways; first, we assess how answers change on in-domain, held-out survey questions. Second, we evaluate whether the model's behavior changes in out-of-domain settings (situational scenarios). To this end, we construct a contextualized moral judgment dataset based on Reddit posts and evaluate changes in the model's behavior in text-based adventure games. We demonstrate that our simple approach can not only change the model's answers to in-domain survey questions, but also produces substantial shifts (value alignment) in implicit downstream task behavior.