mlsanityData to 14 Oct 2026

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs

arXiv 2609.33298 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF also fits: LLM preference optimization methods

Abstract (arXiv)

In many settings, post-training need not create the target behavior from scratch: the base model can already produce it, but not reliably. This shifts part of preference alignment from capability acquisition to behavioral expression. We ask how a specified preference is represented in native model computation, what prevents target-supporting computation from reliably dominating generation, and whether this structure can directly guide control. We introduce Residual Competition Maps (RCMs), which map a behavioral preference onto signed causal effects of native residual computation. Across preference domains, RCMs reveal coexisting target-supporting and target-competing effects, input-dependent component roles, and cases where a single native-component intervention reverses the preference outcome. DPO substantially reorganizes these effects and can weaken opposition without guaranteeing its removal. We then propose Direct Hidden-State Alignment (DHSA), which treats inference-time hidden states rather than base-model weights as the direct adaptation space. RCM-guided Causal Activation State Transition (CAST) implements DHSA through local state interventions at a small number of preference-relevant interfaces while freezing the base model. With only 256-16,384 controller parameters, CAST reaches DPO-competitive operating points across three preference domains, can complement DPO-trained models, and can be enabled or removed at inference time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 47 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

33 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
20 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  3. 2023Steering Language Models With Activation Engineeringcited by 922
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  6. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  7. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  8. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  9. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  10. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  11. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  12. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  13. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 57
  14. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  15. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  16. 2024LIONs: An Empirically Optimized Approach to Align Language Modelscited by 7
  17. 2026Activation Steering via Generative Causal Mediationcited by 6
  18. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexitycited by 4
  19. 2025Toward Preference-aligned Large Language Models via Residual-based Model Steeringcited by 3
  20. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space
  2. 2025Alignment is Localized: A Causal Probe into Preference Layers
  3. 2025Why Alignment Must Precede Distillation: A Minimal Working Explanation
  4. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
  5. 2026STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment
  6. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2024Robust Preference Optimization through Reward Model Distillation