mlsanityData to 14 Oct 2026

Noise Injection Systemically Degrades Large Language Model Safety Guardrails

arXiv 2505.13500 ↗First posted 16 May 2025cs.CL cs.AI cs.LGcited by 4

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety guardrails in large language models (LLMs) are a critical component in preventing harmful outputs. Yet, their resilience under perturbation remains poorly understood. In this paper, we investigate the robustness of safety fine-tuning in LLMs by systematically injecting Gaussian noise into model activations. We show across multiple open-weight models that (1) Gaussian noise raises harmful-output rates (p < 0.001) by up to 27%, (2) that deeper safety fine-tuning affords no extra protection, and (3) that chain-of-thought reasoning remains largely intact. The findings reveal critical vulnerabilities in current safety alignment techniques and highlight the potential of reasoning-based and reinforcement learning approaches as promising direction for developing more robust AI safety systems. These results have important implications for real-world deployment of LLMs in safety-critical applications as these results imply that widely-deployed safety tuning methods can fail even without adversarial prompts.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 22 of its 30 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

22 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study14 Jul
  2. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  10. 2024A StrongREJECT for Empty Jailbreakscited by 385
  11. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  12. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
6 more
  1. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  2. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 170
  3. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  4. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  5. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  6. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimizationcited by 1
  2. 2026Overthinking: Amplifying Reasoning Weights to Extract Learned Secretscited by 1
  3. 2026Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generationcited by 0
  4. 2026On the Fragility of Data Attribution When Learning Is Distributedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  2. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  5. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  6. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  7. 2025Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
  8. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons