mlsanityData to 14 Oct 2026

Understanding helpfulness and harmless tension in reward models

arXiv 2606.13209 ↗First posted 11 Jun 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are a key component of reinforcement learning from human feedback, enabling the alignment of large language models with human values such as helpfulness and harmlessness. However, these alignment goals can conflict, leading to interference when optimised jointly. We refer to this conflict as alignment tension and investigate how it manifests in RMs. To this end, we train RMs optimised for helpfulness, harmlessness, and a combined helpful-harmless objective, and compare their behaviours and internal representations. We find that mixed-objective RMs frequently underperform their single-objective counterparts, suggesting substantial interference between alignment goals. Using activation-based interpretability methods, we identify neurons associated with helpfulness and harmlessness and show that they causally promote their corresponding behaviours while often suppressing the opposing one. At the same time, we find that a substantial fraction of neurons are shared across objectives. These shared neurons exert a disproportionate influence on model behaviour and encode incompatible representations. Our findings provide mechanistic evidence for how alignment objectives are represented and interact within RMs, offering new insights into the challenges of multi-objective alignment and motivating the development of more disentangled and controllable alignment methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 48 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

33 of 48 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Interpretable Reward Model via Sparse Autoencoder12 Aug
  2. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencoder1 Jul
  3. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  4. 2025On the Robustness of Reward Models for Language Model Alignment12 May
  5. 2024Interpreting Language Reward Models via Contrastive Explanations25 Nov
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  8. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF9 Oct
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  11. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
11 more
  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  4. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  5. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  6. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  7. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  8. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoningcited by 15
  9. 2024Future You: A Conversation with an AI-Generated Future Self Reduces Anxiety, Negative Emotions, and Increases Future Self-Continuitycited by 9
  10. 2025Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledgecited by 8
  11. 2025Multi-Domain Explainability of Preferencescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment is Localized: A Causal Probe into Preference Layers
  2. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2025Debiasing Reward Models by Representation Learning with Guarantees
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
  6. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Models
  7. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  8. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementary