# A critical look at the current train/test split in machine learning

arXiv: https://arxiv.org/abs/2106.04525 · HTML: https://mlsanity.com/paper/2106.04525/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning](https://mlsanity.com/paper/1805.10002/) (2018)
- [DeepDTA: Deep Drug-Target Binding Affinity Prediction](https://mlsanity.com/paper/1801.10193/) (2018)
- [Adversarial Active Learning for Deep Networks: a Margin Based Approach](https://mlsanity.com/paper/1802.09841/) (2018)
- [Task-Agnostic Meta-Learning for Few-shot Learning](https://mlsanity.com/paper/1805.07722/) (2018)
- [Generative Adversarial Active Learning](https://mlsanity.com/paper/1702.07956/) (2017)
- [Protein-Ligand Scoring with Convolutional Neural Networks](https://mlsanity.com/paper/1612.02751/) (2016)
- [Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need](https://mlsanity.com/paper/1901.11409/) (2019)
- [Atomic Convolutional Networks for Predicting Protein-Ligand Binding Affinity](https://mlsanity.com/paper/1703.10603/) (2017)
- [RelationNet2: Deep Comparison Columns for Few-Shot Learning](https://mlsanity.com/paper/1811.07100/) (2018)

## Built on by (most cited first, 3 of 3)

- [A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates](https://mlsanity.com/paper/2401.03910/) (2024)
- [Two-Phase Segmentation Approach for Accurate Left Ventricle Segmentation in Cardiac MRI using Machine Learning](https://mlsanity.com/paper/2407.20387/) (2024)
- [Survival of the Cheapest: Cost-Aware Hardware Adaptation for Adversarial Robustness](https://mlsanity.com/paper/2409.07609/) (2024)

## Abstract (arXiv)

The randomized or cross-validated split of training and testing sets has been adopted as the gold standard of machine learning for decades. The establishment of these split protocols are based on two assumptions: (i)-fixing the dataset to be eternally static so we could evaluate different machine learning algorithms or models; (ii)-there is a complete set of annotated data available to researchers or industrial practitioners. However, in this article, we intend to take a closer and critical look at the split protocol itself and point out its weakness and limitation, especially for industrial applications. In many real-world problems, we must acknowledge that there are numerous situations where assumption (ii) does not hold. For instance, for interdisciplinary applications like drug discovery, it often requires real lab experiments to annotate data which poses huge costs in both time and financial considerations. In other words, it can be very difficult or even impossible to satisfy assumption (ii). In this article, we intend to access this problem and reiterate the paradigm of active learning, and investigate its potential on solving problems under unconventional train/test split protocols. We further propose a new adaptive active learning architecture (AAL) which involves an adaptation policy, in comparison with the traditional active learning that only unidirectionally adds data points to the training pool. We primarily justify our points by extensively investigating an interdisciplinary drug-protein binding problem. We additionally evaluate AAL on more conventional machine learning benchmarking datasets like CIFAR-10 to demonstrate the generalizability and efficacy of the new framework.