# How You Split Matters: Data Leakage and Subject Characteristics Studies in Longitudinal Brain MRI Analysis

arXiv: https://arxiv.org/abs/2309.00350 · HTML: https://mlsanity.com/paper/2309.00350/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Builds on (most cited first, 5 of 5)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification](https://mlsanity.com/paper/1701.06643/) (2017)
- [Detecting Shortcuts in Medical Images -- A Case Study in Chest X-rays](https://mlsanity.com/paper/2211.04279/) (2022)
- [Is a PET all you need? A multi-modal study for Alzheimer's disease using 3D CNNs](https://mlsanity.com/paper/2207.02094/) (2022)
- [3D Convolutional Neural Networks for Stalled Brain Capillary Detection](https://mlsanity.com/paper/2104.01687/) (2021)

## Built on by (most cited first, 8 of 8)

- [Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets](https://mlsanity.com/paper/2401.14497/) (2024)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [Dataset Diversity Metrics and Impact on Classification Models](https://mlsanity.com/paper/2603.15276/) (2026)
- [A Leakage-Aware Comparative Benchmark of Machine Learning, Deep Learning, and Transformer Models for Reliable Leukemia Detection](https://mlsanity.com/paper/2606.24944/) (2026)
- [3D MRI-Based Alzheimer's Disease Classification Using Multi-Modal 3D CNN with Leakage-Aware Subject-Level Evaluation](https://mlsanity.com/paper/2603.17304/) (2026)
- [Low-Magnification SEM May Suffice: Interpretable Deep Learning for Multi-Scale Fracture-Cause Classification in Zirconia-Toughened Alumina](https://mlsanity.com/paper/2605.29798/) (2026)
- [Monte Carlo Dropout Uncertainty and Entropy-Thresholded Selective Prediction for Architecture-Agnostic Brain Tumor MRI Triage](https://mlsanity.com/paper/2607.16317/) (2026)
- [Patient-Level, Leakage-Aware Deep Learning for Cross-Center Periapical Radiograph Classification](https://mlsanity.com/paper/2609.14703/) (2026)

## Abstract (arXiv)

Deep learning models have revolutionized the field of medical image analysis, offering significant promise for improved diagnostics and patient care. However, their performance can be misleadingly optimistic due to a hidden pitfall called 'data leakage'. In this study, we investigate data leakage in 3D medical imaging, specifically using 3D Convolutional Neural Networks (CNNs) for brain MRI analysis. While 3D CNNs appear less prone to leakage than 2D counterparts, improper data splitting during cross-validation (CV) can still pose issues, especially with longitudinal imaging data containing repeated scans from the same subject. We explore the impact of different data splitting strategies on model performance for longitudinal brain MRI analysis and identify potential data leakage concerns. GradCAM visualization helps reveal shortcuts in CNN models caused by identity confounding, where the model learns to identify subjects along with diagnostic features. Our findings, consistent with prior research, underscore the importance of subject-wise splitting and evaluating our model further on hold-out data from different subjects to ensure the integrity and reliability of deep learning models in medical image analysis.