mlsanityData to 7 Oct 2026

The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features

arXiv 2509.12934 ↗First posted 16 Sep 2025cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically demonstrate that this mechanism is expressive enough to approximate the behavioral shifts of post-training processes. We then apply FSRL to preference optimization and perform a causal analysis of the learned policy. Our analysis reveals a crucial insight: the model learns to reward stylistic presentation as a proxy for quality, disproportionately relying on features related to style and formatting over those tied to alignment concepts like honesty. By effectively optimizing the preference objective, FSRL serves as a transparent proxy for observing the alignment process. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 48 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

34 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  11. 2022Toy Models of Superpositioncited by 928
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
20 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2023Steering Language Models With Activation Engineeringcited by 824
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  8. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  9. 2024BatchTopK Sparse Autoencoderscited by 126
  10. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  11. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
  12. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  13. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  14. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  15. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  16. 2025Steering Large Language Model Activations in Sparse Spacescited by 58
  17. 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
  18. 2025Automating Steering for Safe Multimodal Large Language Modelscited by 17
  19. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoderscited by 11
  20. 2025Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  2. 2026Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Datacited by 0
  3. 2026Values as Style: Disentangling Values from Semantics with One-Way Mixing for Low-Damage LLM Steeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment is Localized: A Causal Probe into Preference Layers
  2. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
  3. 2026Mechanistic Analysis of Alignment Algorithms in Language Models
  4. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  5. 2024SEAL: Systematic Error Analysis for Value ALignment
  6. 2025Preference Learning for AI Alignment: a Causal Perspective
  7. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  8. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models