# Scaffold Splits Overestimate Virtual Screening Performance

arXiv: https://arxiv.org/abs/2406.00873 · HTML: https://mlsanity.com/paper/2406.00873/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 41 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 2 of 2)

- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)

## Built on by (most cited first, 7 of 7)

- [Tree of Attributes Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.11201/) (2024)
- [Clever Materials: When Models Identify Good Materials for the Wrong Reasons](https://mlsanity.com/paper/2602.17730/) (2026)
- [ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics](https://mlsanity.com/paper/2510.05482/) (2025)
- [A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Era](https://mlsanity.com/paper/2604.16586/) (2026)
- [A standard transformer and attention with linear biases for molecular conformer generation](https://mlsanity.com/paper/2506.19834/) (2025)
- [Benchmarking graph-based models for in-silico toxicity prediction in drug discovery](https://mlsanity.com/paper/2609.37555/) (2026)
- [ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations](https://mlsanity.com/paper/2609.38538/) (2026)

## Abstract (arXiv)

Virtual Screening (VS) of vast compound libraries guided by Artificial Intelligence (AI) models is a highly productive approach to early drug discovery. Data splitting is crucial for better benchmarking of such AI models. Traditional random data splits produce similar molecules between training and test sets, conflicting with the reality of VS libraries which mostly contain structurally distinct compounds. Scaffold split, grouping molecules by shared core structure, is widely considered to reflect this real-world scenario. However, here we show that the scaffold split also overestimates VS performance. The reason is that molecules with different chemical scaffolds are often similar, which hence introduces unrealistically high similarities between training molecules and test molecules following a scaffold split. Our study examined three representative AI models on 60 NCI-60 datasets, each with approximately 30,000 to 50,000 molecules tested on a different cancer cell line. Each dataset was split with three methods: scaffold, Butina clustering and the more accurate Uniform Manifold Approximation and Projection (UMAP) clustering. Regardless of the model, model performance is much worse with UMAP splits from the results of the 2100 models trained and evaluated for each algorithm and split. These robust results demonstrate the need for more realistic data splits to tune, compare, and select models for VS. For the same reason, avoiding the scaffold split is also recommended for other molecular property prediction problems. The code to reproduce these results is available at https://github.com/ScaffoldSplitsOverestimateVS