# From Clever Hans to Scientific Discovery: Interpreting EEG Foundational Transformers with LRP

arXiv: https://arxiv.org/abs/2605.11885 · HTML: https://mlsanity.com/paper/2605.11885/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 65 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals](https://mlsanity.com/paper/2409.00101/) (2024)
- [Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking](https://mlsanity.com/paper/2409.19407/) (2024)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data](https://mlsanity.com/paper/2206.11417/) (2022)
- [Interpretable Deep Neural Networks for Single-Trial EEG Classification](https://mlsanity.com/paper/1604.08201/) (2016)
- [Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation](https://mlsanity.com/paper/1904.08771/) (2019)
- [Analyzing Neuroimaging Data Through Recurrent Deep Learning Models](https://mlsanity.com/paper/1810.09945/) (2018)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [EEG Foundation Challenge: From Cross-Task to Cross-Subject EEG Decoding](https://mlsanity.com/paper/2506.19141/) (2025)
- [EEG-FM-Compass: Progress, Benchmarking, and Future Directions for EEG Foundation Models](https://mlsanity.com/paper/2601.17883/) (2026)
- [Health system learning achieves generalist neuroimaging models](https://mlsanity.com/paper/2511.18640/) (2025)

## Built on by (most cited first, 1 of 1)

- [EEG-PRISM: Physiologically-Grounded Interpretability of Predictions by EEG Foundation Models](https://mlsanity.com/paper/2608.13676/) (2026)

## Abstract (arXiv)

Emerging foundation models (FMs) in electroencephalography (EEG) promise a path to scale deep learning in diagnostics and brain-computer interfaces despite data scarcity, yet their opaque nature remains a barrier to wider adoption. We investigate attention-aware Layer-wise relevance propagation (LRP) as a post-hoc attribution method for EEG-FMs, extending LRP's use on convolutional neural network (CNN)-based EEG models to the Transformer architectures that current FMs are based on. We find that LRP can both verify EEG-FM decisions and surface novel, biologically plausible hypotheses from them. In motor imagery, it unmasks 'Clever Hans' behavior where models prioritize task correlated ocular signals over the intended motor correlates. In a naturalistic paradigm for affect prediction, it reveals a recurring reliance on a central electrode cluster, suggesting a candidate sensorimotor signature of arousal. Though heatmap interpretation remains ambiguous in this complex domain, the results position LRP as a tool for both verification and exploration of EEG-FMs, a role that will grow in both importance and discovery potential as the underlying models mature.