# The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

arXiv: https://arxiv.org/abs/2108.05386 · HTML: https://mlsanity.com/paper/2108.05386/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge](https://mlsanity.com/paper/1612.08012/) (2016)
- [DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification](https://mlsanity.com/paper/1801.09555/) (2018)
- [An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classification](https://mlsanity.com/paper/1806.00712/) (2018)
- [Highly accurate model for prediction of lung nodule malignancy with CT scans](https://mlsanity.com/paper/1802.01756/) (2018)
- [Lung Nodule Classification using Deep Local-Global Networks](https://mlsanity.com/paper/1904.10126/) (2019)
- [DeepLung: 3D Deep Convolutional Nets for Automated Pulmonary Nodule Detection and Classification](https://mlsanity.com/paper/1709.05538/) (2017)

## Built on by (most cited first, 3 of 3)

- [Reducing Annotation Need in Self-Explanatory Models for Lung Nodule Diagnosis](https://mlsanity.com/paper/2206.13608/) (2022)
- [cRedAnno+: Annotation Exploitation in Self-Explanatory Lung Nodule Diagnosis](https://mlsanity.com/paper/2210.16097/) (2022)
- [MERA: Multimodal and Multiscale Self-Explanatory Model with Considerably Reduced Annotation for Lung Nodule Diagnosis](https://mlsanity.com/paper/2504.19357/) (2025)

## Abstract (arXiv)

Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the state-of-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work.