mlsanityData to 7 Oct 2026

AI Security Leaderboard: Methodology, Results and Minimal Standard

arXiv 2608.03070 ↗First posted 4 Aug 2026cs.CR cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The AI Security Leaderboard is an independent benchmark that ranks the safeguards of frontier AI models from least to most secure. It tests models against the FAR.AI Minimal Standard for Safeguards, which represents a minimum bar for security: meeting it does not guarantee a secure model, but failing to meet it guarantees a lack of state-of-the-art security. Version 1.0 covers severe misuse requests across chemical, biological, radiological, nuclear, and explosive (CBRNE) threats and offensive cybersecurity. In this report, we tested four leading models for universal jailbreaks in the context of this minimal standard, and found more than a hundredfold difference in security. Claude Fable 5 and GPT-5.6 Sol held against every attack we ran, with no universal jailbreak found; we estimate they would likely cost more than 14,200 to jailbreak, if it is possible with this methodology at all. Meanwhile, we found hundreds of universal jailbreaks for Grok 4.5 and Gemini 3.1 Pro; each broke for under 300, with universal jailbreaks in Grok's weakest domain, cybersecurity, accessible for as little as 24. The gap is fixable: every weakness we found belongs to a known class of attack that already has a defense deployed in production models. The leaderboard will be updated on a rolling basis as new models are released, and the evaluation methodology and Minimal Standard will be periodically revised to take into account the latest capabilities and the state-of-the-art in safeguards. The leaderboard is available at leaderboard.far.ai.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

15 of 21 references.

References · 15

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  4. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  5. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  6. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  7. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  8. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  9. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  10. 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 26
  11. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  12. 2026Boundary Point Jailbreaking of Black-Box LLMscited by 7
3 more
  1. 2025The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Modelscited by 2
  2. 2026Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attackscited by 2
  3. 2026Jailbroken Frontier Models Retain Their Capabilitiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.