# Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

arXiv: https://arxiv.org/abs/2409.10053 · HTML: https://mlsanity.com/paper/2409.10053/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Personas as a Way to Model Truthfulness in Language Models](https://mlsanity.com/paper/2310.18168/) (2023)
- [A Language Model's Guide Through Latent Space](https://mlsanity.com/paper/2402.14433/) (2024)

## Built on by (most cited first, 23 of 23)

- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [Multi-Attribute Steering of Language Models via Targeted Intervention](https://mlsanity.com/paper/2502.12446/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Spherical Steering: Geometry-Aware Activation Rotation for Language Models](https://mlsanity.com/paper/2602.08169/) (2026)
- [ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment](https://mlsanity.com/paper/2602.17560/) (2026)
- [Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning](https://mlsanity.com/paper/2510.01032/) (2025)
- [GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs](https://mlsanity.com/paper/2507.18043/) (2025)
- [Prototype-Based Dynamic Steering for Large Language Models](https://mlsanity.com/paper/2510.05498/) (2025)
- [Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection](https://mlsanity.com/paper/2601.19375/) (2026)
- [Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention](https://mlsanity.com/paper/2605.05892/) (2026)
- [Probe-Free Low-Rank Activation Intervention](https://mlsanity.com/paper/2502.04043/) (2025)
- [Minimizing Collateral Damage in Activation Steering](https://mlsanity.com/paper/2605.01167/) (2026)
- [Steering Language Models Before They Speak: Logit-Level Interventions](https://mlsanity.com/paper/2601.10960/) (2026)
- [Self-Improving Model Steering](https://mlsanity.com/paper/2507.08967/) (2025)
- [Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior](https://mlsanity.com/paper/2606.08454/) (2026)
- [Conditional Optimal Bridge for Riemannian Activation Steering](https://mlsanity.com/paper/2607.10517/) (2026)
- [Mapping Semantic & Syntactic Relationships with Geometric Rotation](https://mlsanity.com/paper/2510.09790/) (2025)
- [Steering Over-refusals Towards Safety in Retrieval Augmented Generation](https://mlsanity.com/paper/2510.10452/) (2025)
- [Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations](https://mlsanity.com/paper/2602.03237/) (2026)
- [DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment](https://mlsanity.com/paper/2603.21461/) (2026)
- [Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces](https://mlsanity.com/paper/2505.07831/) (2025)
- [Controlling Refusal Behavior of LLMs via Stiefel-Constrained Rotation Steering](https://mlsanity.com/paper/2608.30986/) (2026)
- [GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models](https://mlsanity.com/paper/2609.10658/) (2026)

## Abstract (arXiv)

Activation Editing, which involves directly editting the internal representations of large language models (LLMs) to alter their behaviors and achieve desired properties, has emerged as a promising area of research. Existing works primarily treat LLMs' activations as points in space and modify them by adding steering vectors. However, this approach is limited in its ability to achieve greater performance improvement while maintaining the necessary consistency of activation magnitudes. To overcome these issues, we propose a novel editing method that views activations in terms of their directions and magnitudes. Our method, named Householder Pseudo-Rotation (HPR), mimics the rotation transformation, thus preserving activation norms and resulting in an improved performance on various safety benchmarks.