mlsanityData to 7 Oct 2026

The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment

arXiv 2602.04196 ↗First posted 4 Feb 2026cs.CL cs.LGcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 13 of its 24 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

13 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!21 May

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  6. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  7. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  8. 2024Frontier Models are Capable of In-context Schemingcited by 252
  9. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  10. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  11. 2025Demonstrating specification gaming in reasoning modelscited by 48
  12. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  2. 2026Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems
  3. 2026Alignment Risks from Capability-Seeking RL Training
  4. 2025Agent Safety Alignment via Reinforcement Learning
  5. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  7. 2024AI Risk Management Should Incorporate Both Safety and Security
  8. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning