# Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

arXiv: https://arxiv.org/abs/2610.19817 · HTML: https://mlsanity.com/paper/2610.19817/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models](https://mlsanity.com/paper/2601.10387/) (2026)
- [InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance](https://mlsanity.com/paper/2401.11206/) (2024)
- [Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values](https://mlsanity.com/paper/2311.10766/) (2023)
- [AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint](https://mlsanity.com/paper/2506.07022/) (2025)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2511.00222/) (2025)
- [Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models](https://mlsanity.com/paper/2502.19649/) (2025)
- [The Touch'e23-ValueEval Dataset for Identifying Human Values behind Arguments](https://mlsanity.com/paper/2301.13771/) (2023)
- [Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History](https://mlsanity.com/paper/2508.04826/) (2025)
- [Drift No More? Context Equilibria in Multi-Turn LLM Interactions](https://mlsanity.com/paper/2510.07777/) (2025)
- [Internal Value Alignment in Large Language Models through Controlled Value Vector Activation](https://mlsanity.com/paper/2507.11316/) (2025)
- [Value Drifts: Tracing Value Alignment During LLM Post-Training](https://mlsanity.com/paper/2510.26707/) (2025)
- [Investigating LLMs as Voting Assistants via Contextual Augmentation: A Case Study on the European Parliament Elections 2024](https://mlsanity.com/paper/2407.08495/) (2024)
- [Unintended Harms of Value-Aligned LLMs: Psychological and Empirical Insights](https://mlsanity.com/paper/2506.06404/) (2025)
- [EAVIT: Efficient and Accurate Human Value Identification from Text data via LLMs](https://mlsanity.com/paper/2505.12792/) (2025)
- ["Label from Somewhere": Reflexive Annotating for Situated AI Alignment](https://mlsanity.com/paper/2601.17937/) (2026)
- [Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence](https://mlsanity.com/paper/2604.08169/) (2026)
- [Steering Large Language Models for Machine Translation Personalization](https://mlsanity.com/paper/2505.16612/) (2025)
- [Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments](https://mlsanity.com/paper/2509.06704/) (2025)

## Abstract (arXiv)

Human values shape how we expect language models to behave. Dominant alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.