# Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

arXiv: https://arxiv.org/abs/1606.05201 · HTML: https://mlsanity.com/paper/1606.05201/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2016.

## Built on by (most cited first, 30 of 30)

- [Feature robustness and sex differences in medical imaging: a case study in MRI-based Alzheimer's disease detection](https://mlsanity.com/paper/2204.01737/) (2022)
- [On Leakage in Machine Learning Pipelines](https://mlsanity.com/paper/2311.04179/) (2023)
- [Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research](https://mlsanity.com/paper/2301.08525/) (2023)
- [FetMRQC: a robust quality control system for multi-centric fetal brain MRI](https://mlsanity.com/paper/2311.04780/) (2023)
- [Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex](https://mlsanity.com/paper/2505.15813/) (2025)
- [FetMRQC: Automated Quality Control for fetal brain MRI](https://mlsanity.com/paper/2304.05879/) (2023)
- [On the benefits of self-taught learning for brain decoding](https://mlsanity.com/paper/2209.10099/) (2022)
- [The Gaussian-Linear Hidden Markov model: a Python package](https://mlsanity.com/paper/2312.07151/) (2023)
- [Modeling Large-Scale Walking and Cycling Networks: A Machine Learning Approach Using Mobile Phone and Crowdsourced Data](https://mlsanity.com/paper/2404.00162/) (2024)
- [Why are there many equally good models? An Anatomy of the Rashomon Effect](https://mlsanity.com/paper/2601.06730/) (2026)
- [Statistical Opportunities in Neuroimaging](https://mlsanity.com/paper/2602.12974/) (2026)
- [Meta-learning In-Context Enables Training-Free Cross Subject Brain Decoding](https://mlsanity.com/paper/2604.08537/) (2026)
- [Do Language Models Align with Brains? Prediction Scores Are Not Enough](https://mlsanity.com/paper/2605.14025/) (2026)
- [Learning brain MRI quality control: a multi-factorial generalization problem](https://mlsanity.com/paper/2205.15898/) (2022)
- [Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting](https://mlsanity.com/paper/2308.11197/) (2023)
- [Julearn: an easy-to-use library for leakage-free evaluation and inspection of ML models](https://mlsanity.com/paper/2310.12568/) (2023)
- [Decoding Working-Memory Load During n-Back Task Performance from High Channel NIRS Data](https://mlsanity.com/paper/2312.07546/) (2023)
- [A Machine Learning Approach for Identifying Anatomical Biomarkers of Early Mild Cognitive Impairment](https://mlsanity.com/paper/2407.00040/) (2024)
- [The use of cross validation in the analysis of designed experiments](https://mlsanity.com/paper/2506.14593/) (2025)
- [Rewiring Human Brain Networks via Lightweight Dynamic Connectivity Framework: An EEG-Based Stress Validation](https://mlsanity.com/paper/2511.05505/) (2025)
- [A Reproducible Framework for Bias-Resistant Machine Learning on Small-Sample Neuroimaging Data](https://mlsanity.com/paper/2602.02920/) (2026)
- [Bayesian Inference of Psychometric Variables From Brain and Behavior in Implicit Association Tests](https://mlsanity.com/paper/2603.16741/) (2026)
- [Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding](https://mlsanity.com/paper/2605.00865/) (2026)
- [A Universal Space of Brain Dynamics for Unveiling Cognitive Transitions and Individual Differences](https://mlsanity.com/paper/2605.02936/) (2026)
- [Heterogeneous Neural Predictivity from Language Models During Naturalistic Comprehension](https://mlsanity.com/paper/2606.26880/) (2026)
- [Benchmarking External Generalization of SPD Matrix Learning for Resting-State fMRI Connectome Prediction](https://mlsanity.com/paper/2608.30418/) (2026)
- [Best Practices in EEG Analysis: Preprocessing, Modeling, and Machine Learning](https://mlsanity.com/paper/2609.36609/) (2026)
- [Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoders](https://mlsanity.com/paper/2609.37642/) (2026)
- [Hybrid Machine Learning-Assisted Raman Spectroscopy with Generative Feature Augmentation for Pharmaceutical Identification](https://mlsanity.com/paper/2610.02224/) (2026)
- [Video-Time Priors for EEG-fNIRS Emotion Regression on Familiar Videos](https://mlsanity.com/paper/2610.03618/) (2026)

## Abstract (arXiv)

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets --anatomical and functional MRI and MEG-- and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.