mlsanityData to 14 Oct 2026

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

arXiv 2603.21461 ↗First posted 23 Mar 2026cs.LG cs.AI cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Preference alignment is usually achieved by weight-updating training on preference data, which adds substantial alignment-stage compute and provides limited mechanistic visibility. We propose Dynamic SAE Steering for Preference Alignment (DSPA), an inference-time method that makes sparse autoencoder (SAE) steering prompt-conditional. From preference triples, DSPA computes a conditional-difference map linking prompt features to generation-control features; during decoding, it modifies only token-active latents, without base-model weight updates. Across Gemma-2-2B/9B and Qwen3-8B, DSPA improves MT-Bench and is competitive on AlpacaEval while preserving multiple-choice accuracy. Under restricted preference data, DSPA remains robust and can rival the two-stage RAHF-SCIT pipeline while requiring up to 4.47× fewer alignment-stage FLOPs. Finally, we audit the SAE features DSPA modifies, finding that preference directions are dominated by discourse and stylistic signals, and provide theory clarifying the conditional-difference map estimate and when top-k ablation is principled.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 42 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

25 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation13 Jan
  2. 2025The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features16 Sep
  3. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  4. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods29 Jan
  5. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2022Toy Models of Superpositioncited by 971
  11. 2024Scaling and evaluating sparse autoencoderscited by 553
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
7 more
  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  2. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  3. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  4. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  5. 2025Steering Large Language Model Activations in Sparse Spacescited by 58
  6. 2024Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspectivecited by 23
  7. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
  2. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models
  3. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
  4. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  5. 2025Toward Preference-aligned Large Language Models via Residual-based Model Steering
  6. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selection