mlsanityData to 7 Oct 2026

Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

arXiv 2409.10053 ↗First posted 16 Sep 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Activation Editing, which involves directly editting the internal representations of large language models (LLMs) to alter their behaviors and achieve desired properties, has emerged as a promising area of research. Existing works primarily treat LLMs' activations as points in space and modify them by adding steering vectors. However, this approach is limited in its ability to achieve greater performance improvement while maintaining the necessary consistency of activation magnitudes. To overcome these issues, we propose a novel editing method that views activations in terms of their directions and magnitudes. Our method, named Householder Pseudo-Rotation (HPR), mimics the rotation transformation, thus preserving activation norms and resulting in an improved performance on various safety benchmarks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

27 of 39 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
15 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  6. 2022A Survey on In-context Learningcited by 717
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  11. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  12. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  13. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  14. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 46
  15. 2024A Language Model's Guide Through Latent Spacecited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 12.

23 papers.

Sort:

By area:

  1. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  2. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  3. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  4. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 17
  5. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 15
  6. 2025Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoningcited by 8
  7. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  8. 2025Probe-Free Low-Rank Activation Interventioncited by 5
  9. 2025Prototype-Based Dynamic Steering for Large Language Modelscited by 4
  10. 2026Selective Steering: Norm-Preserving Control Through Discriminative Layer Selectioncited by 4
  11. 2026Minimizing Collateral Damage in Activation Steeringcited by 4
  12. 2026Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Interventioncited by 4
11 more
  1. 2025Self-Improving Model Steeringcited by 3
  2. 2026Steering Language Models Before They Speak: Logit-Level Interventionscited by 3
  3. 2025Mapping Semantic & Syntactic Relationships with Geometric Rotationcited by 1
  4. 2025Steering Over-refusals Towards Safety in Retrieval Augmented Generationcited by 1
  5. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  6. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  7. 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behaviorcited by 1
  8. 2026Conditional Optimal Bridge for Riemannian Activation Steeringcited by 1
  9. 2025Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spacescited by 0
  10. 2026Controlling Refusal Behavior of LLMs via Stiefel-Constrained Rotation Steeringcited by 0
  11. 2026GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Modelscited by 0