mlsanityData to 14 Oct 2026

Safety Training May Persist Through Helpfulness Optimization in LLM Agents

arXiv 2603.02229 ↗First posted 13 Feb 2026cs.LG cs.CLcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Safety post-training has been studied extensively in single-step "chat" settings where safety typically refers to refusing harmful requests. We study an "agentic" (i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We investigate the effects of using direct preference optimization (DPO) to optimize safety and/or helpfulness on the ToolEmu agentic benchmark. First, we find that safety training largely persists through subsequent helpfulness training. Second, we find a consistent negative linear correlation (R2 = 0.77) between safety and helpfulness when considering all training configurations together. Even post-training on both metrics simultaneously simply results in another point on the same trend line rather than yielding a "best of both worlds" strategy, despite the presence of such strategies in our dataset. Overall, our findings underscore the need for a better understanding of post-training.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 27 of its 38 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

27 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents19 Oct
  2. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety18 Oct
  3. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation19 Aug
  4. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
11 more
  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  4. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  5. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  6. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  7. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  8. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  9. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  10. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  11. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 17

Similar, not cited

Close in wording, no citation either way

  1. 2026The Cold-Start Safety Gap in LLM Agents
  2. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  3. 2026The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
  4. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
  5. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
  6. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
  7. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach
  8. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agents