mlsanityData to 14 Oct 2026

Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

arXiv 2610.19817 ↗First posted 14 Oct 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human values shape how we expect language models to behave. Dominant alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 52 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

30 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Value Drifts: Tracing Value Alignment During LLM Post-Training30 Oct
  2. 2025Internal Value Alignment in Large Language Models through Controlled Value Vector Activation15 Jul
  3. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications21 Mar

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  8. 2020Artificial Intelligence, Values and Alignmentcited by 448
  9. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  10. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  11. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 90
  12. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
15 more
  1. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 50
  2. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  3. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 43
  4. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 39
  5. 2025Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Modelscited by 34
  6. 2023The Touch'e23-ValueEval Dataset for Identifying Human Values behind Argumentscited by 28
  7. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 26
  8. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 22
  9. 2024Investigating LLMs as Voting Assistants via Contextual Augmentation: A Case Study on the European Parliament Elections 2024cited by 7
  10. 2025Unintended Harms of Value-Aligned LLMs: Psychological and Empirical Insightscited by 7
  11. 2025EAVIT: Efficient and Accurate Human Value Identification from Text data via LLMscited by 5
  12. 2026"Label from Somewhere": Reflexive Annotating for Situated AI Alignmentcited by 4
  13. 2026Activation Steering for Aligned Open-ended Generation without Sacrificing Coherencecited by 4
  14. 2025Steering Large Language Models for Machine Translation Personalizationcited by 3
  15. 2025Will Annotators Disagree? Identifying Subjectivity in Value-Laden Argumentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective
  2. 2026Steering Geometry: Validating Human Value Geometry in LLM Steering Space
  3. 2026VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
  4. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  5. 2023Heterogeneous Value Alignment Evaluation for Large Language Models
  6. 2025Beyond Single-Sentence Prompts: Upgrading Value Alignment Benchmarks with Dialogues and Stories
  7. 2025Cognitive models can reveal interpretable value trade-offs in language models
  8. 2026Adaptive Multi-Value Control in LLMs via Causal Activation Steering