# Contrastive Covariance Subspaces for Activation Steering

arXiv: https://arxiv.org/abs/2610.14025 · HTML: https://mlsanity.com/paper/2610.14025/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 29 of 29)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Improving Activation Steering in Language Models with Mean-Centring](https://mlsanity.com/paper/2312.03813/) (2023)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Emotion Concepts and their Function in a Large Language Model](https://mlsanity.com/paper/2604.07729/) (2026)
- [Spectral Editing of Activations for Large Language Model Alignment](https://mlsanity.com/paper/2405.09719/) (2024)
- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [Representation Surgery: Theory and Practice of Affine Steering](https://mlsanity.com/paper/2402.09631/) (2024)
- [BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data](https://mlsanity.com/paper/2410.16491/) (2024)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models](https://mlsanity.com/paper/2410.02298/) (2024)
- [Improved Representation Steering for Language Models](https://mlsanity.com/paper/2505.20809/) (2025)
- [Parts of Speech-Grounded Subspaces in Vision-Language Models](https://mlsanity.com/paper/2305.14053/) (2023)
- [Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions](https://mlsanity.com/paper/2605.10664/) (2026)
- [Do Personality Traits Interfere? Geometric Limitations of Steering in Large Language Models](https://mlsanity.com/paper/2602.15847/) (2026)
- [Faithful Bi-Directional Model Steering via Distribution Matching and Distributed Interchange Interventions](https://mlsanity.com/paper/2602.05234/) (2026)
- [Conceptors for Semantic Steering](https://mlsanity.com/paper/2605.04980/) (2026)
- [A Low-Rank Subspace Analysis of LLM Interventions](https://mlsanity.com/paper/2606.14388/) (2026)

## Abstract (arXiv)

Activation steering controls large language model behavior by adding a direction to hidden activations during inference. For steering to work well, that direction must be a pure signal for the target concept: any confound entangled with it, such as an incidental topic or writing style, is injected into the residual stream alongside the concept itself and degrades the generated text. We introduce Contrastive Covariance Steering (CCS), a training-free method that eliminates this confound variation. CCS projects steering vectors to a low-dimensional subspace obtained in closed form by solving a contrastive variance objective, which retains the directions along which the target concept varies while suppressing directions shared with the other classes. Across three LLMs and four steering domains, CCS consistently beats all baseline steering methods in all 12 model-domain settings. The largest relative improvement occurs on AxBench and Emotion where we outperform the next strongest steering baseline by 48 to 70% and 13 to 27%, respectively. Furthermore, CCS outperforms naive difference-of-means steering by up to 151%, showcasing the importance of deconfounding concepts. A toy experiment verifies our motivation on why: planting a topic nuisance into the training data reduces unprojected steering by 46%, whereas CCS loses only 12% and recovers most of the gap to an oracle that explicitly deletes the planted direction. A human study with 23 raters on 296 generations reproduces the LLM judge's ranking. Our extensive ablation verifies the importance of our method's components and robustness to the number of nuisance classes. Finally, we extend our method to multi-turn conversations, where CCS sustains both concept expression and fluency, matching system prompt performance as other steering methods drift.