mlsanityData to 14 Oct 2026

Spectral Editing of Activations for Large Language Model Alignment

arXiv 2405.09719 ↗First posted 15 May 2024cs.CL cs.AI cs.LGcited by 56

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) often exhibit undesirable behaviours, such as generating untruthful or biased content. Editing their internal representations has been shown to be effective in mitigating such behaviours on top of the existing alignment methods. We propose a novel inference-time editing method, namely spectral editing of activations (SEA), to project the input representations into directions with maximal covariance with the positive demonstrations (e.g., truthful) while minimising covariance with the negative demonstrations (e.g., hallucinated). We also extend our method to non-linear editing using feature functions. We run extensive experiments on benchmarks concerning truthfulness and bias with six open-source LLMs of different sizes and model families. The results demonstrate the superiority of SEA in effectiveness, generalisation to similar tasks, as well as computation and data efficiency. We also show that SEA editing only has a limited negative impact on other model capabilities.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 49 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

41 of 49 references.

References · 41

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Mistral 7Bcited by 3,164
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2024Mixtral of Expertscited by 1,823
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
29 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2023Steering Language Models With Activation Engineeringcited by 922
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  7. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  10. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  11. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  13. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  14. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  15. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  16. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  17. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  18. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
  19. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  20. 2024Representation Surgery: Theory and Practice of Affine Steeringcited by 50
  21. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  22. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  23. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 34
  24. 2023A Trip Towards Fairness: Bias and De-Biasing in Large Language Modelscited by 32
  25. 2023Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projectioncited by 25
  26. 2023Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Modelscited by 13
  27. 2023Erasure of Unaligned Attributes from Neural Representationscited by 6
  28. 2023A Joint Matrix Factorization Analysis of Multilingual Representationscited by 5
  29. 2022Understanding Domain Learning in Language Models Through Subpopulation Analysiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 20.

56 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  2. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  3. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 63
  4. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  5. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  6. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  7. 2025Representation Bending for Large Language Model Safetycited by 30
  8. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 29
  9. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 21
  10. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  11. 2025COSMIC: Generalized Refusal Direction Identification in LLM Activationscited by 19
  12. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 18
38 more
  1. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 18
  2. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 17
  3. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  4. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  5. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 12
  6. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 12
  7. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  8. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMscited by 10
  9. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  10. 2026Spectral Attention Steering for Prompt Highlightingcited by 8
  11. 2025Inference-time Alignment in Continuous Spacecited by 7
  12. 2025Steering Protein Language Modelscited by 7
  13. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  14. 2025Prototype-Based Dynamic Steering for Large Language Modelscited by 6
  15. 2026The Unreasonable Effectiveness of Text Embedding Interpolation for Continuous Image Steeringcited by 6
  16. 2025DISCO: Disentangled Communication Steering for Large Language Modelscited by 5
  17. 2025Efficient LLM Moderation with Multi-Layer Latent Prototypescited by 4
  18. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  19. 2026Steering Language Models Before They Speak: Logit-Level Interventionscited by 4
  20. 2025Self-Improving Model Steeringcited by 3
  21. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
  22. 2026On the Context Sensitivity of LLM Moral Judgmentcited by 3
  23. 2026Don't Lose Focus: Activation Steering via Key-Orthogonal Projectionscited by 3
  24. 2025A Markov Categorical Framework for Language Modelingcited by 2
  25. 2026PRISM-Delta: Differential Subspace Steering for Prompt Highlighting in Large Language Modelscited by 2
  26. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 2
  27. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  28. 2025Inference-Time Intervention in Large Language Models for Reliable Requirement Verificationcited by 1
  29. 2025Detoxification of Large Language Models through Output-layer Fusion with a Calibration Modelcited by 1
  30. 2025Iterative Multilingual Spectral Attribute Erasurecited by 1
  31. 2026Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimescited by 1
  32. 2025MASteer: Multi-Agent Adaptive Steer Strategy for End-to-End LLM Trustworthiness Repaircited by 0
  33. 2025Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answerscited by 0
  34. 2026Aligning Recommendations with User Popularity Preferencescited by 0
  35. 2026A Systematic Study of Training-Free Methods for Trustworthy Large Language Modelscited by 0
  36. 2026Pref-CTRL: Preference Driven LLM Alignment using Representation Editingcited by 0
  37. 2026Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steeringcited by 0
  38. 2026TUX: Measuring Human--AI Tacit Understandingcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Large Language Models with Representation Editing: A Control Perspective
  2. 2024Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective
  3. 2026Beyond the Covariance Trap: Unlocking Generalization in Same-Subject Knowledge Editing for Large Language Models
  4. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
  5. 2025EAMET: Robust Massive Model Editing via Embedding Alignment Optimization
  6. 2025Aligning Language Models with Real-time Knowledge Editing
  7. 2025Benchmarking and Rethinking Knowledge Editing for Large Language Models
  8. 2025EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing