# SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models

arXiv: https://arxiv.org/abs/2509.03487 · HTML: https://mlsanity.com/paper/2509.03487/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 7 of 7)

- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding](https://mlsanity.com/paper/2408.08252/) (2024)
- [AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies](https://mlsanity.com/paper/2407.17436/) (2024)
- [Proteina: Scaling Flow-based Protein Structure Generative Models](https://mlsanity.com/paper/2503.00710/) (2025)
- [DPLM-2: A Multimodal Diffusion Protein Language Model](https://mlsanity.com/paper/2410.13782/) (2024)
- [AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration](https://mlsanity.com/paper/2503.15754/) (2025)
- [Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization](https://mlsanity.com/paper/2507.10923/) (2025)

## Built on by (most cited first, 5 of 5)

- [Generative AI for Biosciences: Emerging Threats and Roadmap to Biosecurity](https://mlsanity.com/paper/2510.15975/) (2025)
- [A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models](https://mlsanity.com/paper/2608.02684/) (2026)
- [VFUSE: Virulent Feature Understanding with Sparse autoEncoders](https://mlsanity.com/paper/2606.10080/) (2026)
- [SafeBench-Seq: A Homology-Clustered, CPU-Only Baseline for Protein Hazard Screening with Physicochemical/Composition Features and Cluster-Aware Confidence Intervals](https://mlsanity.com/paper/2512.17527/) (2025)
- [Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance](https://mlsanity.com/paper/2609.16213/) (2026)

## Abstract (arXiv)

Proteins play crucial roles in almost all biological processes. The advancement of deep learning has greatly accelerated the development of protein foundation models, leading to significant successes in protein understanding and design. However, the lack of systematic red-teaming for these models has raised serious concerns about their potential misuse, such as generating proteins with biological safety risks. This paper introduces SafeProtein, the first red-teaming framework designed for protein foundation models to the best of our knowledge. SafeProtein combines multimodal prompt engineering and heuristic beam search to systematically design red-teaming methods and conduct tests on protein foundation models. We also curated SafeProtein-Bench, which includes a manually constructed red-teaming benchmark dataset and a comprehensive evaluation protocol. SafeProtein achieved continuous jailbreaks on state-of-the-art protein foundation models (up to 70% attack success rate for ESM3), revealing potential biological safety risks in current protein foundation models and providing insights for the development of robust security protection technologies for frontier models. The codes will be made publicly available at https://github.com/jigang-fan/SafeProtein.