# PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

arXiv: https://arxiv.org/abs/2606.24081 · HTML: https://mlsanity.com/paper/2606.24081/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Placed in

- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 29 of 29)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Self-correcting LLM-controlled Diffusion Models](https://mlsanity.com/paper/2311.16090/) (2023)
- [SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning](https://mlsanity.com/paper/2409.05556/) (2024)
- [Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models](https://mlsanity.com/paper/2404.02928/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [Perception-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2408.10848/) (2024)
- [Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey](https://mlsanity.com/paper/2411.09259/) (2024)
- [Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models](https://mlsanity.com/paper/2408.00523/) (2024)
- [DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization](https://mlsanity.com/paper/2408.11071/) (2024)
- [FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts](https://mlsanity.com/paper/2502.21059/) (2025)
- [T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model](https://mlsanity.com/paper/2510.22300/) (2025)
- [HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Models](https://mlsanity.com/paper/2408.13896/) (2024)
- [Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoning](https://mlsanity.com/paper/2503.17987/) (2025)
- [Harnessing LLM to Attack LLM-Guarded Text-to-Image Models](https://mlsanity.com/paper/2312.07130/) (2023)
- [Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters](https://mlsanity.com/paper/2604.01888/) (2026)
- [GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models](https://mlsanity.com/paper/2506.10047/) (2025)
- [Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking](https://mlsanity.com/paper/2602.24009/) (2026)

## Abstract (arXiv)

As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More importantly, T2I jailbreak evaluation is not a single prompt-level test, but a pipeline-level problem shaped by multiple stages, including prompt transformation, image generation, safety filtering, and multimodal judging. This makes results across papers difficult to reliably reproduce and fairly compare. To bridge this gap, we propose PixJail, a self-evolving paper-to-pipeline agent framework for reproducible T2I jailbreak evaluation. Given a T2I jailbreak paper and optional reference code, PixJail rapidly constructs a paper-specific attack module and a runnable evaluation pipeline under a unified contract, while faithfully reproducing the original experimental results. PixJail further maintains a memory bank that stores paper digests, attack evolution patterns, reusable templates, failure cases, and versioned artifacts, enabling future reproduction efforts to reuse prior experience. We reproduce eleven representative T2I jailbreak methods, including both code-available and code-unavailable papers. Under their original settings, our framework accurately recovers prior results with minimal error (2.1% average, 0% median). We hope that PixJail can serve as a unified foundation for future T2I jailbreak reproduction and evaluation, significantly reducing manual effort.