mlsanityData to 14 Oct 2026

Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods

arXiv 2401.16332 ↗First posted 29 Jan 2024cs.CL cs.AIcited by 28

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, representation engineering, a method which alters the model's behavior via changing its representations post-training, was shown to be effective in aligning LLMs (Zou et al., 2023a). Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. First, we find that under the conditions of our framework, alignment can be guaranteed with representation engineering, and at the same time that helpfulness is harmed in the process. Second, we show that helpfulness is harmed quadratically with the norm of the representation engineering vector, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of representation engineering for alignment.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 72 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

43 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  2. 2023Fundamental Limitations of Alignment in Large Language Models19 Apr
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2016Concrete Problems in AI Safetycited by 1,642
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
28 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2023Steering Language Models With Activation Engineeringcited by 922
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2023In-Context Learning Creates Task Vectorscited by 312
  13. 2021Unsolved Problems in ML Safetycited by 292
  14. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  15. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  16. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  17. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  18. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  19. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  20. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 68
  21. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 46
  22. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  23. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 37
  24. 2023Downstream Trade-offs of a Family of Text Watermarkscited by 14
  25. 2023Advances in apparent conceptual physics reasoning in GPT-4cited by 12
  26. 2021Automatically Exposing Problems with Neural Dialog Modelscited by 5
  27. 2023Can Large Language Models Change User Preference Adversarially?cited by 5
  28. 2020On the Ethics of Building AI in a Responsible Mannercited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 12.

28 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  3. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  4. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  5. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  6. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  7. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  8. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  9. 2024Large language models can consistently generate high-quality content for election disinformation operationscited by 14
  10. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 10
  11. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  12. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
16 more
  1. 2025Efficient LLM Moderation with Multi-Layer Latent Prototypescited by 4
  2. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 4
  3. 2026AI Alignment Breaks at the Edgecited by 4
  4. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  5. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  6. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
  7. 2026Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilitiescited by 1
  8. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agentscited by 1
  9. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  10. 2026Selective Safety Steering via Value-Filtered Decodingcited by 0
  11. 2026Multi-Adapter Representation Interventions via Energy Calibrationcited by 0
  12. 2026Temporal Preference Concepts and their Functions in a Large Language Modelcited by 0
  13. 2026Towards the Readability of LLM-Generated Codes through Multitask Representation Engineeringcited by 0
  14. 2026Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposurecited by 0
  15. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0
  16. 2026Which Knowledge Should LLMs Trust? Uncovering a Generalizable Steering Direction Across Knowledge Conflictscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2023The Alignment Problem in Context
  3. 2023Large Language Model Alignment: A Survey
  4. 2025Alignment is Localized: A Causal Probe into Preference Layers
  5. 2024DeAL: Decoding-time Alignment for Large Language Models
  6. 2024Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
  7. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong
  8. 2024Unintended Impacts of LLM Alignment on Global Representation