# Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning

arXiv: https://arxiv.org/abs/2204.08682 · HTML: https://mlsanity.com/paper/2204.08682/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 103 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Molecule Attention Transformer](https://mlsanity.com/paper/2002.08264/) (2020)
- [Learning to Extend Molecular Scaffolds with Structural Motifs](https://mlsanity.com/paper/2103.03864/) (2021)
- [SMILES Enumeration as Data Augmentation for Neural Network Modeling of Molecules](https://mlsanity.com/paper/1703.07076/) (2017)
- [Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention](https://mlsanity.com/paper/2103.00213/) (2021)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [In silico generation of novel, drug-like chemical matter using the LSTM neural network](https://mlsanity.com/paper/1712.07449/) (2017)
- [Accelerating Prototype-Based Drug Discovery using Conditional Diversity Networks](https://mlsanity.com/paper/1804.02668/) (2018)

## Built on by (most cited first, 3 of 3)

- [Difficulty in chirality recognition for Transformer architectures learning chemical structures from string](https://mlsanity.com/paper/2303.11593/) (2023)
- [A novel molecule generative model of VAE combined with Transformer for unseen structure generation](https://mlsanity.com/paper/2402.11950/) (2024)
- [From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models](https://mlsanity.com/paper/2605.09949/) (2026)

## Abstract (arXiv)

Adverse events are a serious issue in drug development and many prediction methods using machine learning have been developed. The random split cross-validation is the de facto standard for model building and evaluation in machine learning, but care should be taken in adverse event prediction because this approach does not match to the real-world situation. The time split, which uses the time axis, is considered suitable for real-world prediction. However, the differences in model performance obtained using the time and random splits are not clear due to the lack of the comparable studies. To understand the differences, we compared the model performance between the time and random splits using nine types of compound information as input, eight adverse events as targets, and six machine learning algorithms. The random split showed higher area under the curve values than did the time split for six of eight targets. The chemical spaces of the training and test datasets of the time split were similar, suggesting that the concept of applicability domain is insufficient to explain the differences derived from the splitting. The area under the curve differences were smaller for the protein interaction than for the other datasets. Subsequent detailed analyses suggested the danger of confounding in the use of knowledge-based information in the time split. These findings indicate the importance of understanding the differences between the time and random splits in adverse event prediction and strongly suggest that appropriate use of the splitting strategies and interpretation of results are necessary for the real-world prediction of adverse events. We provide analysis code and datasets used in the present study (https://github.com/mizuno-group/AE_prediction).