Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 39 of its 57 references that are in the map. First posted on arXiv 27 Feb 2026.
Builds on
39 of 57 references.
Closest ancestors · 31
Same direction, newest first
- 2025EquaCode: A Multi-Strategy Jailbreak Approach for Large Language Models via Equation Solving and Code Completion29 Dec
- 2025TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations5 Dec
- 2025"To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios20 Nov
- 2025The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning24 Oct
- 2025Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency24 Oct
- 2025Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge22 Sep
- 2025A Simple and Efficient Jailbreak Method Exploiting LLMs' Helpfulness17 Sep
- 2025Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience25 Aug
- 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs22 Aug
- 2025PUZZLED: Jailbreaking LLMs through Word-Based Puzzles2 Aug
- 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models7 Jul
- 2025Alphabet Index Mapping: Jailbreaking LLMs through Semantic Dissimilarity15 Jun
19 more
- 2025Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking8 Apr
- 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods24 Feb
- 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos19 Feb
- 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language13 Feb
- 2024SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage19 Dec
- 2024You Know What I'm Saying: Jailbreak Attack via Implicit Reference4 Oct
- 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
- 2024FlipAttack: Jailbreak LLMs via Flipping2 Oct
- 2024Reasoning as a Weapon: Adaptive Dual-Path Jailbreak Attack on Large Language Models23 Jul
- 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response22 May
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
- 2024A StrongREJECT for Empty Jailbreaks15 Feb
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
- 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
- 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
Other references · 8
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
- 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
- 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 29
- 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
- 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
- 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.