# Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignment

arXiv: https://arxiv.org/abs/2609.05800 · HTML: https://mlsanity.com/paper/2609.05800/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)
- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization](https://mlsanity.com/paper/2310.03708/) (2023)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors](https://mlsanity.com/paper/2410.12299/) (2024)
- [Extending Activation Steering to Broad Skills and Multiple Behaviours](https://mlsanity.com/paper/2403.05767/) (2024)
- [Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering](https://mlsanity.com/paper/2410.16314/) (2024)
- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models](https://mlsanity.com/paper/2507.01915/) (2025)
- [Internal Value Alignment in Large Language Models through Controlled Value Vector Activation](https://mlsanity.com/paper/2507.11316/) (2025)
- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [Dynamically Scaled Activation Steering](https://mlsanity.com/paper/2512.03661/) (2025)
- [OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment](https://mlsanity.com/paper/2509.24610/) (2025)
- [Compositional Steering of Large Language Models with Steering Tokens](https://mlsanity.com/paper/2601.05062/) (2026)
- [Minimizing Collateral Damage in Activation Steering](https://mlsanity.com/paper/2605.01167/) (2026)
- [PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization](https://mlsanity.com/paper/2507.16679/) (2025)
- [VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models](https://mlsanity.com/paper/2602.03160/) (2026)
- [Controllable Value Alignment in Large Language Models through Neuron-Level Editing](https://mlsanity.com/paper/2602.07356/) (2026)

## Built on by (most cited first, 1 of 1)

- [Adaptive Multi-Value Control in LLMs via Causal Activation Steering](https://mlsanity.com/paper/2609.30405/) (2026)

## Abstract (arXiv)

Activation steering controls LLM behavior at inference time by adding learned directions to hidden states, but existing methods handle one concept at a time. Pluralistic alignment, where different stakeholders need different value emphases, requires steering multiple dimensions simultaneously. We show that naive steering produces substantial spillover: the effect intended for one value leaks into others. This parallels the treatment-versus-spillover decomposition in causal inference. We trace spillover to geometric entanglement of steering directions, captured by their Gram matrix, and derive a zero-cost correction from an activation-norm-penalized objective that decouples each direction's contribution exactly. Our end-to-end pipeline requires no fine-tuning, no reward model, and no manual prompt engineering: given only domain questions, it automatically discovers value dimensions, extracts directions, diagnoses entanglement, and applies corrected steering. On climate discourse, the correction improves the net steering effect from +5.9% to +14.0%, validated over 100,000 pairwise judgments.