mlsanityData to 7 Oct 2026

Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense

arXiv 2510.01088 ↗First posted 1 Oct 2025cs.AIcited by 4

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Ensuring Large Language Model (LLM) safety remains challenging due to the absence of universal standards and reliable content validators, making it difficult to obtain effective training signals. We discover that aligned models already possess robust internal safety beliefs: they consistently produce high-confidence refusals to harmful requests while exhibiting high entropy when generating potentially dangerous content. This entropy gap reveals an untapped signal--models intrinsically "know" when to refuse. We introduce Safety Instincts Reinforcement Learning (SIRL), which transforms this internal confidence into a self-generated reward signal, eliminating dependence on external validators or human annotations. SIRL teaches models to trust their safety instincts by reinforcing low-entropy refusal behaviors. Evaluated on Llama and Qwen models, SIRL maintains 89%+ Defense Success Rates (DSRs) against 20+ jailbreak methods, from static prompts to adaptive attacks. Using only 15,000 unlabeled prompts, SIRL surpasses resource-intensive supervised methods while preserving performance on mathematics, coding, and conversation benchmarks. Our work demonstrates that effective alignment can emerge from within, paving the way for more autonomous and robust AI safety mechanisms that scale without extensive human oversight.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 52 of its 65 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

52 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Does Refusal Training in LLMs Generalize to the Past Tense?16 Jul
  2. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons20 Jun

Other references · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
38 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2025Group Sequence Policy Optimizationcited by 788
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  12. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  13. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  14. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  15. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  16. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  17. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  18. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  19. 2023Detecting Language Model Attacks with Perplexitycited by 411
  20. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  21. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  22. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  23. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  24. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  25. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  26. 2025Learning to Reason without External Rewardscited by 199
  27. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  28. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  29. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  30. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  31. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  32. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  33. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  34. 2018Supervising strong learners by amplifying weak expertscited by 139
  35. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  36. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  37. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  38. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multilingual Safety Alignment via Self-Distillationcited by 1
  2. 2026Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregationcited by 0
  3. 2026Refusing Intent, Not Form: Wrapper-Based Intent-Group Supervision for LLM Safetycited by 0
  4. 2026Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Modelscited by 0