# The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

arXiv: https://arxiv.org/abs/2404.15319 · HTML: https://mlsanity.com/paper/2404.15319/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 117 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [MOABB: Trustworthy algorithm benchmarking for BCIs](https://mlsanity.com/paper/1805.06427/) (2018)
- [EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain-Machine Interfaces](https://mlsanity.com/paper/2006.00622/) (2020)
- [Data augmentation for learning predictive models on EEG: a systematic comparison](https://mlsanity.com/paper/2206.14483/) (2022)
- [Unraveling the Hidden Environmental Impacts of AI Solutions for Environment](https://mlsanity.com/paper/2110.11822/) (2021)
- [Benchopt: Reproducible, efficient and collaborative optimization benchmarks](https://mlsanity.com/paper/2206.13424/) (2022)
- [2021 BEETL Competition: Advancing Transfer Learning for Subject Independence & Heterogenous EEG Data Sets](https://mlsanity.com/paper/2202.12950/) (2022)
- [Toward reliable signals decoding for electroencephalogram: A benchmark study to EEGNeX](https://mlsanity.com/paper/2207.12369/) (2022)
- [A Plug&Play P300 BCI Using Information Geometry](https://mlsanity.com/paper/1409.0107/) (2014)
- [Learning from Label Proportions in Brain-Computer Interfaces: Online Unsupervised Learning with Guarantees](https://mlsanity.com/paper/1701.07213/) (2017)
- [Evaluating the structure of cognitive tasks with transfer learning](https://mlsanity.com/paper/2308.02408/) (2023)
- [Comparative evaluation of state-of-the-art algorithms for SSVEP-based BCIs](https://mlsanity.com/paper/1602.00904/) (2016)
- [Building Brain Invaders: EEG data of an experimental validation](https://mlsanity.com/paper/1905.05182/) (2019)
- [Functional connectivity ensemble method to enhance BCI performance (FUCONE)](https://mlsanity.com/paper/2111.03122/) (2021)
- [Minimizing subject-dependent calibration for BCI with Riemannian transfer learning](https://mlsanity.com/paper/2111.12071/) (2021)
- [Frequency Recognition in SSVEP-based BCI using Multiset Canonical Correlation Analysis](https://mlsanity.com/paper/1308.5609/) (2013)
- [Brain Invaders Adaptive versus Non-Adaptive P300 Brain-Computer Interface dataset](https://mlsanity.com/paper/1904.09111/) (2019)
- [Classification of BCI-EEG based on augmented covariance matrix](https://mlsanity.com/paper/2302.04508/) (2023)
- [Pseudo-online framework for BCI evaluation: A MOABB perspective](https://mlsanity.com/paper/2308.11656/) (2023)
- [UMM: Unsupervised Mean-difference Maximization](https://mlsanity.com/paper/2306.11830/) (2023)
- [Transfer Learning between Motor Imagery Datasets using Deep Learning -- Validation of Framework and Comparison of Datasets](https://mlsanity.com/paper/2311.16109/) (2023)
- [Comparison of an open-hardware electroencephalography amplifier with medical grade device in brain-computer interface applications](https://mlsanity.com/paper/1606.02438/) (2016)
- [Dataset of an EEG-based BCI experiment in Virtual Reality and on a Personal Computer](https://mlsanity.com/paper/1903.11297/) (2019)
- [Improving P300 Speller performance by means of optimization and machine learning](https://mlsanity.com/paper/2007.06411/) (2020)

## Built on by (most cited first, 29 of 29)

- [Brain-to-Text Decoding: A Non-invasive Approach via Typing](https://mlsanity.com/paper/2502.17480/) (2025)
- [AdaBrain-Bench: Benchmarking Brain Foundation Models for Brain-Computer Interface Applications](https://mlsanity.com/paper/2507.09882/) (2025)
- [SPDIM: Source-Free Unsupervised Conditional and Label Shift Adaptation in EEG](https://mlsanity.com/paper/2411.07249/) (2024)
- [EEG Foundation Challenge: From Cross-Task to Cross-Subject EEG Decoding](https://mlsanity.com/paper/2506.19141/) (2025)
- [DBConformer: Dual-Branch Convolutional Transformer for EEG Decoding](https://mlsanity.com/paper/2506.21140/) (2025)
- [Mitigating Subject Dependency in EEG Decoding with Subject-Specific Low-Rank Adapters](https://mlsanity.com/paper/2510.08059/) (2025)
- [Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction](https://mlsanity.com/paper/2603.16281/) (2026)
- [Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings](https://mlsanity.com/paper/2608.18114/) (2026)
- [Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendations](https://mlsanity.com/paper/2405.19345/) (2024)
- [MVCNet: Multi-View Contrastive Network for Motor Imagery Classification](https://mlsanity.com/paper/2502.17482/) (2025)
- [SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities](https://mlsanity.com/paper/2407.11676/) (2024)
- [HEEGNet: Hyperbolic Embeddings for EEG](https://mlsanity.com/paper/2601.03322/) (2026)
- [FedSPDnet: Geometry-Aware Federated Deep Learning with SPDnet](https://mlsanity.com/paper/2604.22494/) (2026)
- [Elliptical Wishart distributions: information geometry, maximum likelihood estimator, performance analysis and statistical learning](https://mlsanity.com/paper/2411.02726/) (2024)
- [Riemannian Geometry-Preserving Variational Autoencoder for MI-BCI Data Augmentation](https://mlsanity.com/paper/2603.10563/) (2026)
- [Stacked LoRA for Subject-Adaptive EEG Foundation Models in Motor Imagery Decoding](https://mlsanity.com/paper/2607.03094/) (2026)
- [Towards Generalizable Learning Models for EEG-Based Identification of Pain Perception](https://mlsanity.com/paper/2508.11691/) (2025)
- [Explicit modelling of subject dependency in BCI decoding](https://mlsanity.com/paper/2509.23247/) (2025)
- [Rethinking Generalized BCIs: Benchmarking 340,000+ Unique Algorithmic Configurations for EEG Mental Command Decoding](https://mlsanity.com/paper/2512.02978/) (2025)
- [DANCE: Detect and Classify Events in EEG](https://mlsanity.com/paper/2605.10688/) (2026)
- [Escaping the Capacity Ceiling: Routing on the Stiefel Manifold for Bilinear SPD Layers](https://mlsanity.com/paper/2605.31043/) (2026)
- [Average Rankings Mask Per-Subject Optimality: A Friedman-Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders](https://mlsanity.com/paper/2606.24394/) (2026)
- [Riemannian Geometry for Pre-trained Language Model Embeddings](https://mlsanity.com/paper/2607.07047/) (2026)
- [Subject-Level Heterogeneity in EEG Motor Imagery Decoding: A Large-Scale Benchmark and Portfolio-Based Reduction of the Search Space](https://mlsanity.com/paper/2607.22778/) (2026)
- [Bayesian Complete-Pooling in Cross-Subject Classification for Motor Imagery Electroencephalogram](https://mlsanity.com/paper/2607.22980/) (2026)
- [Matched-Input Estimates Differ in Sign Across Architectures: Auditing EEG Foundation Models on Motor Imagery](https://mlsanity.com/paper/2609.23924/) (2026)
- [Neuron-Level Architecture Growth: A Controlled Evaluation for EEG Time-Series Decoding](https://mlsanity.com/paper/2609.33880/) (2026)
- [EEG-Fusion: Failure-Informed Source-Free Expert Routing for Robust Motor Imagery EEG Decoding](https://mlsanity.com/paper/2609.33962/) (2026)
- [Benchmarking Label-Revealed Online Updates for EEG BCI Decoding](https://mlsanity.com/paper/2610.07420/) (2026)

## Abstract (arXiv)

Objective. This study conduct an extensive Brain-computer interfaces (BCI) reproducibility analysis on open electroencephalography datasets, aiming to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The need for such benchmark lies in the rapid industrial progress that has given rise to undisclosed proprietary solutions. Furthermore, the scientific literature is dense, often featuring challenging-to-reproduce evaluations, making comparisons between existing approaches arduous. Approach. Within an open framework, 30 machine learning pipelines (separated into raw signal: 11, Riemannian: 13, deep learning: 6) are meticulously re-implemented and evaluated across 36 publicly available datasets, including motor imagery (14), P300 (15), and SSVEP (7). The analysis incorporates statistical meta-analysis techniques for results assessment, encompassing execution time and environmental impact considerations. Main results. The study yields principled and robust results applicable to various BCI paradigms, emphasizing motor imagery, P300, and SSVEP. Notably, Riemannian approaches utilizing spatial covariance matrices exhibit superior performance, underscoring the necessity for significant data volumes to achieve competitive outcomes with deep learning techniques. The comprehensive results are openly accessible, paving the way for future research to further enhance reproducibility in the BCI domain. Significance. The significance of this study lies in its contribution to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility in driving advancements within the field.