# PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis

arXiv: https://arxiv.org/abs/2408.10609 · HTML: https://mlsanity.com/paper/2408.10609/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 68 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 10 of 10)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests](https://mlsanity.com/paper/1509.02237/) (2015)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Modelling Cellular Perturbations with the Sparse Additive Mechanism Shift Variational Autoencoder](https://mlsanity.com/paper/2311.02794/) (2023)
- [Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling](https://mlsanity.com/paper/2211.03553/) (2022)

## Built on by (most cited first, 30 of 35)

- [TxPert: Leveraging Biochemical Relationships for Out-of-Distribution Transcriptomic Perturbation Prediction](https://mlsanity.com/paper/2505.14919/) (2025)
- [Diversity by Design: Addressing Mode Collapse Improves scRNA-seq Perturbation Modeling on Well-Calibrated Metrics](https://mlsanity.com/paper/2506.22641/) (2025)
- [VCWorld: A Biological World Model for Virtual Cell Simulation](https://mlsanity.com/paper/2512.00306/) (2025)
- [Lingshu-Cell: A generative cellular world model for transcriptome modeling toward virtual cells](https://mlsanity.com/paper/2603.25240/) (2026)
- [Unlocking LLM Creativity in Science through Analogical Reasoning](https://mlsanity.com/paper/2605.11258/) (2026)
- [STRAND: Sequence-Conditioned Transport for Single-Cell Perturbations](https://mlsanity.com/paper/2602.10156/) (2026)
- [HarmonyCell: Automating Single-Cell Perturbation Modeling under Semantic and Distribution Shifts](https://mlsanity.com/paper/2603.01396/) (2026)
- [No Foundations without Foundations -- Why semi-mechanistic models are essential for regulatory biology](https://mlsanity.com/paper/2501.19178/) (2025)
- [MapPFN: Learning Causal Perturbation Maps in Context](https://mlsanity.com/paper/2601.21092/) (2026)
- [AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents](https://mlsanity.com/paper/2605.10876/) (2026)
- [Large Language Models for Zero-shot Inference of Causal Structures in Biology](https://mlsanity.com/paper/2503.04347/) (2025)
- [Modeling Gene Expression Distributional Shifts for Unseen Genetic Perturbations](https://mlsanity.com/paper/2507.02980/) (2025)
- [PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation](https://mlsanity.com/paper/2510.07964/) (2025)
- [Progressive Multi-Agent Reasoning for Biological Perturbation Prediction](https://mlsanity.com/paper/2602.07408/) (2026)
- [PRiMeFlow: Capturing Complex Expression Heterogeneity in Perturbation Response Modelling](https://mlsanity.com/paper/2604.13986/) (2026)
- [CellScientist: From Execution Feedback to Auditable Model-Revision Trajectories for Cellular Perturbation Prediction](https://mlsanity.com/paper/2605.07335/) (2026)
- [Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects](https://mlsanity.com/paper/2605.31522/) (2026)
- [The Metric Picks the Winner: Evaluation Choice Flips Model Rankings for Drug-Response Prediction in Unseen Chemistry](https://mlsanity.com/paper/2606.12639/) (2026)
- [D2R2: Discrete Diffusion with Regulation Reinforcement for Single-Cell Perturbation Prediction](https://mlsanity.com/paper/2608.15288/) (2026)
- [Morphologically Intelligent Perturbation Prediction with FORM](https://mlsanity.com/paper/2510.21337/) (2025)
- [SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention](https://mlsanity.com/paper/2604.16776/) (2026)
- [AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories](https://mlsanity.com/paper/2604.19606/) (2026)
- [CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation](https://mlsanity.com/paper/2605.00930/) (2026)
- [Querying Counterfactuals on Tissue Graphs with Supervised Disentanglement](https://mlsanity.com/paper/2606.08493/) (2026)
- [Knowledge Graphs and Reasoning LLMs for Finding Simple Yet Effective Transcriptomic Perturbation Predictors](https://mlsanity.com/paper/2606.08816/) (2026)
- [Chem2Gen-Bench: Benchmarking Chemical-to-Genetic Translation in Perturbation Response Space](https://mlsanity.com/paper/2606.21109/) (2026)
- [Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap](https://mlsanity.com/paper/2607.04595/) (2026)
- [TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology](https://mlsanity.com/paper/2607.08803/) (2026)
- [PerturbMap: Cross-Context Transfer of Single-Cell Perturbation Responses](https://mlsanity.com/paper/2607.28090/) (2026)
- [Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction](https://mlsanity.com/paper/2608.00152/) (2026)

## Abstract (arXiv)

We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery.