Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
The Adversarial Humanities Benchmark (AHB) evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms. Starting from harmful tasks drawn from MLCommons AILuminate, the benchmark rewrites the same objectives through humanities-style transformations while preserving intent. This extends literature on Adversarial Poetry and Adversarial Tales from single jailbreak operators to a broader benchmark family of stylistic obfuscation and goal concealment. In the benchmark results reported here, the original attacks record 3.84% attack success rate (ASR), while transformed methods range from 36.8% to 65.0%, yielding 55.75% overall ASR across 31 frontier models. Under a European Union AI Act Code-of-Practice-inspired systemic-risk lens, Chemical, biological, radiological and nuclear (CBRN) is the highest bucket. Taken together, this lack of stylistic robustness suggests that current safety techniques suffer from weak generalization: deep understanding of 'non-maleficence' remains a central unresolved problem in frontier model safety.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 31 of its 38 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
31 of 38 references.
Closest ancestors · 1
Same direction, newest first
Other references · 30
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
- 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
18 more
- 2024A StrongREJECT for Empty Jailbreakscited by 352
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
- 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
- 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
- 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
- 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
- 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
- 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
- 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
- 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
- 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
- 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
- 2025AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommonscited by 31
- 2025Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Modelscited by 14
- 2024Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injectioncited by 8
- 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Modelscited by 5
- 2025From Adversarial Poetry to Adversarial Tales: An Interpretability Research Agendacited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.