# Improving Activation Steering in Language Models with Mean-Centring

arXiv: https://arxiv.org/abs/2312.03813 · HTML: https://mlsanity.com/paper/2312.03813/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 8 of 8)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)

## Built on by (most cited first, 30 of 68)

- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [Defending Against Unforeseen Failure Modes with Latent Adversarial Training](https://mlsanity.com/paper/2403.05030/) (2024)
- [Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories](https://mlsanity.com/paper/2406.00034/) (2024)
- [Steering Without Side Effects: Improving Post-Deployment Control of Language Models](https://mlsanity.com/paper/2406.15518/) (2024)
- [Robustifying Safety-Aligned Large Language Models through Clean Data Curation](https://mlsanity.com/paper/2405.19358/) (2024)
- [Extending Activation Steering to Broad Skills and Multiple Behaviours](https://mlsanity.com/paper/2403.05767/) (2024)
- [Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering](https://mlsanity.com/paper/2410.16314/) (2024)
- [SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models](https://mlsanity.com/paper/2502.11356/) (2025)
- [Representation Bending for Large Language Model Safety](https://mlsanity.com/paper/2504.01550/) (2025)
- [Controlling Large Language Model Agents with Entropic Activation Steering](https://mlsanity.com/paper/2406.00244/) (2024)
- [SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models](https://mlsanity.com/paper/2505.16188/) (2025)
- [To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models](https://mlsanity.com/paper/2510.13290/) (2025)
- [Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods](https://mlsanity.com/paper/2401.16332/) (2024)
- [Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution](https://mlsanity.com/paper/2410.00153/) (2024)
- [Towards Understanding Distilled Reasoning Models: A Representational Approach](https://mlsanity.com/paper/2503.03730/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [A Causal Explainable Guardrails for Large Language Models](https://mlsanity.com/paper/2405.04160/) (2024)
- [Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach](https://mlsanity.com/paper/2501.11041/) (2025)
- [BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation](https://mlsanity.com/paper/2510.10157/) (2025)
- [Risk Assessment Framework for Code LLMs via Leveraging Internal States](https://mlsanity.com/paper/2504.14640/) (2025)
- [Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering](https://mlsanity.com/paper/2505.15038/) (2025)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [Efficient and accurate steering of Large Language Models through attention-guided feature learning](https://mlsanity.com/paper/2602.00333/) (2026)
- [Causality is Key for Interpretability Claims to Generalise](https://mlsanity.com/paper/2602.16698/) (2026)
- [IF-GUIDE: Influence Function-Guided Detoxification of LLMs](https://mlsanity.com/paper/2506.01790/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Designing Role Vectors to Improve LLM Inference Behaviour](https://mlsanity.com/paper/2502.12055/) (2025)

## Abstract (arXiv)

Recent work in activation steering has demonstrated the potential to better control the outputs of Large Language Models (LLMs), but it involves finding steering vectors. This is difficult because engineers do not typically know how features are represented in these models. We seek to address this issue by applying the idea of mean-centring to steering vectors. We find that taking the average of activations associated with a target dataset, and then subtracting the mean of all training activations, results in effective steering vectors. We test this method on a variety of models on natural language tasks by steering away from generating toxic text, and steering the completion of a story towards a target genre. We also apply mean-centring to extract function vectors, more effectively triggering the execution of a range of natural language tasks by a significant margin (compared to previous baselines). This suggests that mean-centring can be used to easily improve the effectiveness of activation steering in a wide range of contexts.