# Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

arXiv: https://arxiv.org/abs/2508.04899 · HTML: https://mlsanity.com/paper/2508.04899/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 49 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 3 of 3)

- [Neonatal seizure detection from raw multi-channel EEG using a fully convolutional architecture](https://mlsanity.com/paper/2105.13854/) (2021)
- [Scaling convolutional neural networks achieves expert-level seizure detection in neonatal EEG](https://mlsanity.com/paper/2405.09911/) (2024)
- [Ensemble learning using individual neonatal data for seizure detection](https://mlsanity.com/paper/2204.07043/) (2022)

## Built on by (most cited first, 2 of 2)

- [Automated interictal epileptic spike detection from simple and noisy annotations in MEG data](https://mlsanity.com/paper/2510.21596/) (2025)
- [Expert-level detection of low-voltage intervals in EEG as a robust biomarker of neonatal encephalopathy](https://mlsanity.com/paper/2610.12680/) (2026)

## Abstract (arXiv)

Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inconsistent and biased metrics, hindering model comparability and interpretability. Expert-level claims about AI performance are frequently made without rigorous validation, raising concerns about their reliability. This study aims to systematically evaluate common performance metrics and propose best practices tailored to the specific challenges of neonatal seizure detection. Using real and synthetic seizure annotations, we assessed standard performance metrics, consensus strategies, and human-expert level equivalence tests under varying class imbalance, inter-rater agreement, and number of raters. Matthews and Pearson's correlation coefficients outperformed the area under the curve in reflecting performance under class imbalance. Consensus types are sensitive to the number of raters and agreement level among them. Among human-expert level equivalence tests, the multi-rater Turing test using Fleiss k best captured expert-level AI performance. We recommend reporting: (1) at least one balanced metric, (2) Sensitivity, specificity, PPV and NPV, (3) Multi-rater Turing test results using Fleiss k, and (4) All the above on held-out validation set. This proposed framework provides an important prerequisite to clinical validation by enabling a thorough and honest appraisal of AI methods for neonatal seizure detection.