# Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding

arXiv: https://arxiv.org/abs/2605.00865 · HTML: https://mlsanity.com/paper/2605.00865/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 9 of 9)

- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [MOABB: Trustworthy algorithm benchmarking for BCIs](https://mlsanity.com/paper/1805.06427/) (2018)
- [Transfer Learning for Brain-Computer Interfaces: A Euclidean Space Data Alignment Approach](https://mlsanity.com/paper/1808.05464/) (2018)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [Assessing and tuning brain decoders: cross-validation, caveats, and guidelines](https://mlsanity.com/paper/1606.05201/) (2016)
- [How EEG preprocessing shapes decoding performance](https://mlsanity.com/paper/2410.14453/) (2024)
- [CIPHER: Conformer-based Inference of Phonemes from High-density EEG](https://mlsanity.com/paper/2604.02362/) (2026)

## Abstract (arXiv)

We tested whether auditory-evoked EEG supports subject-independent five-vowel perception decoding when trial identity, model identity, prediction provenance, and participant-level inference are controlled within a single benchmark. We reconstructed Study 2 event tables from OpenNeuro ds006104 version 1.0.1 and analysed the consonant-vowel pair task. One-to-one marker-stimulus pairing yielded 3,840 independent trials; control-condition selection and artifact rejection retained 1,094 epochs from 16 participants and 61 EEG channels. Thirteen unique implementations were evaluated using leave-one-subject-out testing, with participant metrics reconstructed from 36,102 trial predictions across 33 complete prediction replicas. Random Forest was numerically highest at 21.474% balanced accuracy (95% participant-bootstrap interval, 19.526-23.482%; chance, 20%), but neither its participant-level tests nor any implementation survived correction across the 13-model family. Deep-model performance was close to chance, and several architectures showed substantial seed-dependent variation and low trial-label agreement. An exploratory MDM analysis comprising 9,616 genuine refits across training cohorts of 3-15 participants showed no monotonic performance gain. Within this dataset and protocol, evidence for reliable cross-subject five-vowel decoding is limited. The benchmark provides a reproducible chain from source rows to retained epochs, predictions, participant-level metrics, multiplicity-adjusted inference, and bounded diagnostic analyses.