# Validating the Validation: Reanalyzing a large-scale comparison of Deep Learning and Machine Learning models for bioactivity prediction

arXiv: https://arxiv.org/abs/1905.11681 · HTML: https://mlsanity.com/paper/1905.11681/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 6 of 6)

- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [Transformers for molecular property prediction: Lessons learned from the past five years](https://mlsanity.com/paper/2404.03969/) (2024)
- [Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study](https://mlsanity.com/paper/2209.13492/) (2022)
- [Analysis of Atom-level pretraining with Quantum Mechanics (QM) data for Graph Neural Networks Molecular property models](https://mlsanity.com/paper/2405.14837/) (2024)
- [Building Robust Machine Learning Models for Small Chemical Science Data: The Case of Shear Viscosity](https://mlsanity.com/paper/2208.10784/) (2022)
- [Best practices for machine learning in antibody discovery and development](https://mlsanity.com/paper/2312.08470/) (2023)

## Abstract (arXiv)

Machine learning methods may have the potential to significantly accelerate drug discovery. However, the increasing rate of new methodological approaches being published in the literature raises the fundamental question of how models should be benchmarked and validated. We reanalyze the data generated by a recently published large-scale comparison of machine learning models for bioactivity prediction and arrive at a somewhat different conclusion. We show that the performance of support vector machines is competitive with that of deep learning methods. Additionally, using a series of numerical experiments, we question the relevance of area under the receiver operating characteristic curve as a metric in virtual screening, and instead suggest that area under the precision-recall curve should be used in conjunction with the receiver operating characteristic. Our numerical experiments also highlight challenges in estimating the uncertainty in model performance via scaffold-split nested cross validation.