# Large EEG-U-Transformer for Time-Step Level Detection Without Pre-Training

arXiv: https://arxiv.org/abs/2504.00336 · HTML: https://mlsanity.com/paper/2504.00336/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2106.06716/) (2021)
- [Transformer-based Spatial-Temporal Feature Learning for EEG Decoding](https://mlsanity.com/paper/2106.11170/) (2021)
- [U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging](https://mlsanity.com/paper/1910.11162/) (2019)
- [The Temple University Hospital Seizure Detection Corpus](https://mlsanity.com/paper/1801.08085/) (2018)
- [U-Net Transformer: Self and Cross Attention for Medical Image Segmentation](https://mlsanity.com/paper/2103.06104/) (2021)
- [Self-Supervised Graph Neural Networks for Improved Electroencephalographic Seizure Analysis](https://mlsanity.com/paper/2104.08336/) (2021)
- [Transformer Convolutional Neural Networks for Automated Artifact Detection in Scalp EEG](https://mlsanity.com/paper/2208.02405/) (2022)
- [Self-supervised EEG Representation Learning for Automatic Sleep Staging](https://mlsanity.com/paper/2110.15278/) (2021)
- [BIOT: Cross-data Biosignal Learning in the Wild](https://mlsanity.com/paper/2305.10351/) (2023)
- [SzCORE: A Seizure Community Open-source Research Evaluation framework for the validation of EEG-based automated seizure detection algorithms](https://mlsanity.com/paper/2402.13005/) (2024)
- [Avoiding Post-Processing with Event-Based Detection in Biomedical Signals](https://mlsanity.com/paper/2209.11007/) (2022)
- [A Novel U-Net Architecture for Denoising of Real-world Noise Corrupted Phonocardiogram Signal](https://mlsanity.com/paper/2310.00216/) (2023)

## Built on by (most cited first, 2 of 2)

- [NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces](https://mlsanity.com/paper/2605.14698/) (2026)
- [LookAroundNet: Extending Temporal Context with Transformers for Clinically Viable EEG Seizure Detection](https://mlsanity.com/paper/2601.06016/) (2026)

## Abstract (arXiv)

Electroencephalography (EEG) reflects the brain's functional state, making it a crucial tool for diverse detection applications like seizure detection and sleep stage classification. While deep learning-based approaches have recently shown promise for automated detection, traditional models are often constrained by limited learnable parameters and only achieve modest performance. In contrast, large foundation models showed improved capabilities by scaling up the model size, but required extensive time-consuming pre-training. Moreover, both types of existing methods require complex and redundant post-processing pipelines to convert discrete labels to continuous annotations. In this work, based on the multi-scale nature of EEG events, we propose a simple U-shaped model to efficiently learn representations by capturing both local and global features using convolution and self-attentive modules for sequence-to-sequence modeling. Compared to other window-level classification models, our method directly outputs predictions at the time-step level, eliminating redundant overlapping inferences. Beyond sequence-to-sequence modeling, the architecture naturally extends to window-level classification by incorporating an attention-pooling layer. Such a paradigm shift and model design demonstrated promising efficiency improvement, cross-subject generalization, and state-of-the-art performance in various time-step and window-level classification tasks in the experiment. More impressively, our model showed the capability to be scaled up to the same level as existing large foundation models that have been extensively pre-trained over diverse datasets and outperforms them by solely using the downstream fine-tuning dataset. Our model won 1st place in the 2025 "seizure detection challenge" organized in the International Conference on Artificial Intelligence in Epilepsy and Other Neurological Disorders.