# Extracting Latent Steering Vectors from Pretrained Language Models

arXiv: https://arxiv.org/abs/2205.05124 · HTML: https://mlsanity.com/paper/2205.05124/

## Key facts

- Cited by 225 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 10 May 2022.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Residual Energy-Based Models for Text Generation](https://mlsanity.com/paper/2004.11714/) (2020)
- [A Probabilistic Formulation of Unsupervised Text Style Transfer](https://mlsanity.com/paper/2002.03912/) (2020)
- [Tailor: Generating and Perturbing Text with Semantic Controls](https://mlsanity.com/paper/2107.07150/) (2021)
- [StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer](https://mlsanity.com/paper/2104.05196/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [Plug and Play Autoencoders for Conditional Text Generation](https://mlsanity.com/paper/2010.02983/) (2020)

## Built on by (most cited first, 30 of 225)

- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [MemOS: A Memory OS for AI System](https://mlsanity.com/paper/2507.03724/) (2025)
- [Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models](https://mlsanity.com/paper/2405.16833/) (2024)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [Aligner: Efficient Alignment by Learning to Correct](https://mlsanity.com/paper/2402.02416/) (2024)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories](https://mlsanity.com/paper/2406.00034/) (2024)
- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models](https://mlsanity.com/paper/2505.22101/) (2025)
- [Understanding Reasoning in Thinking Language Models via Steering Vectors](https://mlsanity.com/paper/2506.18167/) (2025)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors](https://mlsanity.com/paper/2410.12299/) (2024)
- [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024)
- [Spectral Editing of Activations for Large Language Model Alignment](https://mlsanity.com/paper/2405.09719/) (2024)
- [Representation Surgery: Theory and Practice of Affine Steering](https://mlsanity.com/paper/2402.09631/) (2024)
- [Understanding (Un)Reliability of Steering Vectors in Language Models](https://mlsanity.com/paper/2505.22637/) (2025)
- [Steering Without Side Effects: Improving Post-Deployment Control of Language Models](https://mlsanity.com/paper/2406.15518/) (2024)

## Abstract (arXiv)

Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. We hypothesize that the information needed to steer the model to generate a target sentence is already encoded within the model. Accordingly, we explore a different approach altogether: extracting latent vectors directly from pretrained language model decoders without fine-tuning. Experiments show that there exist steering vectors, which, when added to the hidden states of the language model, generate a target sentence nearly perfectly (> 99 BLEU) for English sentences from a variety of domains. We show that vector arithmetic can be used for unsupervised sentiment transfer on the Yelp sentiment benchmark, with performance comparable to models tailored to this task. We find that distances between steering vectors reflect sentence similarity when evaluated on a textual similarity benchmark (STS-B), outperforming pooled hidden states of models. Finally, we present an analysis of the intrinsic properties of the steering vectors. Taken together, our results suggest that frozen LMs can be effectively controlled through their latent steering space.