# ProteinBench: A Holistic Evaluation of Protein Foundation Models

arXiv: https://arxiv.org/abs/2409.06744 · HTML: https://mlsanity.com/paper/2409.06744/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 77 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem](https://mlsanity.com/paper/2206.04119/) (2022)
- [Practical and Asymptotically Exact Conditional Sampling in Diffusion Models](https://mlsanity.com/paper/2306.17775/) (2023)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [SE(3)-Stochastic Flow Matching for Protein Backbone Generation](https://mlsanity.com/paper/2310.02391/) (2023)
- [Protein structure generation via folding diffusion](https://mlsanity.com/paper/2209.15611/) (2022)
- [AlphaFold Meets Flow Matching for Generating Protein Ensembles](https://mlsanity.com/paper/2402.04845/) (2024)
- [Diffusion Language Models Are Versatile Protein Learners](https://mlsanity.com/paper/2402.18567/) (2024)
- [PiFold: Toward effective and efficient protein inverse folding](https://mlsanity.com/paper/2209.12643/) (2022)
- [Conditional Antibody Design as 3D Equivariant Graph Translation](https://mlsanity.com/paper/2208.06073/) (2022)
- [Structure-informed Language Models Are Protein Designers](https://mlsanity.com/paper/2302.01649/) (2023)
- [EigenFold: Generative Protein Structure Prediction with Diffusion Models](https://mlsanity.com/paper/2304.02198/) (2023)
- [Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds](https://mlsanity.com/paper/2301.12485/) (2023)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [End-to-End Full-Atom Antibody Design](https://mlsanity.com/paper/2302.00203/) (2023)
- [Protein Conformation Generation via Force-Guided SE(3) Diffusion Models](https://mlsanity.com/paper/2403.14088/) (2024)
- [Improved motif-scaffolding with SE(3) flow matching](https://mlsanity.com/paper/2401.04082/) (2024)
- [Antigen-Specific Antibody Design via Direct Energy-based Preference Optimization](https://mlsanity.com/paper/2403.16576/) (2024)
- [Str2Str: A Score-based Framework for Zero-shot Protein Conformation Sampling](https://mlsanity.com/paper/2306.03117/) (2023)
- [PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design](https://mlsanity.com/paper/2312.00080/) (2023)

## Built on by (most cited first, 25 of 25)

- [Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review](https://mlsanity.com/paper/2501.09685/) (2025)
- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Design](https://mlsanity.com/paper/2503.19300/) (2025)
- [Elucidating the Design Space of Multimodal Protein Language Models](https://mlsanity.com/paper/2504.11454/) (2025)
- [Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks](https://mlsanity.com/paper/2508.17158/) (2025)
- [An All-Atom Generative Model for Designing Protein Complexes](https://mlsanity.com/paper/2504.13075/) (2025)
- [ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.00162/) (2025)
- [ConfRover: Simultaneous Modeling of Protein Conformation and Dynamics via Autoregression](https://mlsanity.com/paper/2505.17478/) (2025)
- [Improving Protein Sequence Design through Designability Preference Optimization](https://mlsanity.com/paper/2506.00297/) (2025)
- [Proteo-R1: Reasoning Foundation Models for De Novo Protein Design](https://mlsanity.com/paper/2605.02937/) (2026)
- [PFMBench: Protein Foundation Model Benchmark](https://mlsanity.com/paper/2506.14796/) (2025)
- [Designing Cyclic Peptides via Harmonic SDE with Atom-Bond Modeling](https://mlsanity.com/paper/2505.21452/) (2025)
- [Evaluating Relational Reasoning in LLMs with REL](https://mlsanity.com/paper/2604.12176/) (2026)
- [AffinityFlow: Guided Flows for Antibody Affinity Maturation](https://mlsanity.com/paper/2502.10365/) (2025)
- [SinAE: A Single-Architecture Flow-Matching Autoencoder for Cross-Domain Atomic Systems](https://mlsanity.com/paper/2607.12380/) (2026)
- [Non-Canonical Crosslinks Confound Evolutionary Protein Structure Models](https://mlsanity.com/paper/2503.17368/) (2025)
- [General Protein Pretraining or Domain-Specific Designs? Benchmarking Protein Modeling on Realistic Applications](https://mlsanity.com/paper/2506.02052/) (2025)
- [PepEDiff: Zero-Shot Peptide Binder Design via Protein Embedding Diffusion](https://mlsanity.com/paper/2601.13327/) (2026)
- [BABE: Biology Arena BEnchmark](https://mlsanity.com/paper/2602.05857/) (2026)
- [Demystifying Multimodal Biomolecular Co-design With Intrinsic Geodesic Coupling](https://mlsanity.com/paper/2606.01628/) (2026)
- [TadA-Bench: A Million-Variant Benchmark for Future-Round Discovery Toward Agentic Protein Engineering](https://mlsanity.com/paper/2606.02624/) (2026)
- [Pepti-drift: Scalable Safe-Active Peptide Generation Without Inference-Time Guidance](https://mlsanity.com/paper/2606.27824/) (2026)
- [Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design](https://mlsanity.com/paper/2607.20057/) (2026)
- [Learning Latent Protein Languages for Autoregressive Generation](https://mlsanity.com/paper/2610.03978/) (2026)
- [ProTok: Unified Protein Tokenization via Modular Multi-Codebook Quantization](https://mlsanity.com/paper/2610.18481/) (2026)

## Abstract (arXiv)

Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.