mlsanityData to 14 Oct 2026

Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries

arXiv 2604.00228 ↗First posted 31 Mar 2026cs.CLcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models are trained to refuse harmful requests, but can they accurately predict when they will refuse before responding? We investigate this question through a systematic study where models first predict their refusal behavior, then respond in a fresh context. Across 3754 datapoints spanning 300 requests, we evaluate four frontier models: Claude Sonnet 4, Claude Sonnet 4.5, GPT-5.2, and Llama 3.1 405B. Using signal detection theory (SDT), we find that all models exhibit high introspective sensitivity (d' = 2.4-3.5), but sensitivity drops substantially at safety boundaries. We observe generational improvement within Claude (Sonnet 4.5: 95.7 percent accuracy vs Sonnet 4: 93.0 percent), while GPT-5.2 shows lower accuracy (88.9 percent) with more variable behavior. Llama 405B achieves high sensitivity but exhibits strong refusal bias and poor calibration, resulting in lower overall accuracy (80.0 percent). Topic-wise analysis reveals weapons-related queries are consistently hardest for introspection. Critically, confidence scores provide actionable signal: restricting to high-confidence predictions yields 98.3 percent accuracy for well-calibrated models, enabling practical confidence-based routing for safety-critical deployments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 13 of its 15 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

13 of 15 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2022Red Teaming Language Models with Language Modelscited by 1,026
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2023Do Large Language Models Know What They Don't Know?cited by 243
  12. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
1 more
  1. 2026Emergent Introspective Awareness in Large Language Modelscited by 82

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
  2. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  3. 2026Latent-space Attacks for Refusal Evasion in Language Models
  4. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  5. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
  6. 2023I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Models
  7. 2026Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
  8. 2025Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal