mlsanityData to 14 Oct 2026

TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors

arXiv 2609.05117 ↗First posted 4 Sep 2026cs.CR cs.AIcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts yield the most diverse behaviors, while jailbreaks reveal the largest robustness gaps. Furthermore, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 15 of its 21 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

15 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness4 Feb
  2. 2026Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context25 Jan
  3. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm26 Jun
  4. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Mistral 7Bcited by 3,164
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  7. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  9. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  10. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
  11. 2025SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledgecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
  2. 2026Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs
  3. 2025LLM-Safety Evaluations Lack Robustness
  4. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  5. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
  6. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
  7. 2026Analysing the Safety Pitfalls of Steering Vectors
  8. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions