# BIOT: Cross-data Biosignal Learning in the Wild

arXiv: https://arxiv.org/abs/2305.10351 · HTML: https://mlsanity.com/paper/2305.10351/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 64 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Transformer-based Spatial-Temporal Feature Learning for EEG Decoding](https://mlsanity.com/paper/2106.11170/) (2021)
- [Transformer Convolutional Neural Networks for Automated Artifact Detection in Scalp EEG](https://mlsanity.com/paper/2208.02405/) (2022)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [Improved EEG Event Classification Using Differential Energy](https://mlsanity.com/paper/1801.02477/) (2018)
- [Self-supervised EEG Representation Learning for Automatic Sleep Staging](https://mlsanity.com/paper/2110.15278/) (2021)
- [ManyDG: Many-domain Generalization for Healthcare Applications](https://mlsanity.com/paper/2301.08834/) (2023)
- [ATD: Augmenting CP Tensor Decomposition by Self Supervision](https://mlsanity.com/paper/2106.07900/) (2021)

## Built on by (most cited first, 20 of 20)

- [Large Cognition Model: Towards Pretrained EEG Foundation Model](https://mlsanity.com/paper/2502.17464/) (2025)
- [A Simple Review of EEG Foundation Models: Datasets, Advancements and Future Perspectives](https://mlsanity.com/paper/2504.20069/) (2025)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities](https://mlsanity.com/paper/2510.00032/) (2025)
- [BioSerenity-E1: a self-supervised EEG model for medical applications](https://mlsanity.com/paper/2503.10362/) (2025)
- [LuMamba: Latent Unified Mamba for Electrode Topology-Invariant and Efficient EEG Modeling](https://mlsanity.com/paper/2603.19100/) (2026)
- [Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendations](https://mlsanity.com/paper/2405.19345/) (2024)
- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)
- [Large EEG-U-Transformer for Time-Step Level Detection Without Pre-Training](https://mlsanity.com/paper/2504.00336/) (2025)
- [NeuroTTT: Bridging Pretraining-Downstream Task Misalignment in EEG Foundation Models via Test-Time Training](https://mlsanity.com/paper/2509.26301/) (2025)
- [Federated Few-Shot Learning for Epileptic Seizure Detection Under Privacy Constraints](https://mlsanity.com/paper/2512.13717/) (2025)
- [Towards Unified Multi-task EEG Analysis with Low-Rank Adaptation](https://mlsanity.com/paper/2604.25131/) (2026)
- [CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding](https://mlsanity.com/paper/2607.09543/) (2026)
- [DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations](https://mlsanity.com/paper/2607.11578/) (2026)
- [Interpretable EEG biomarkers with bag-of-waves: Spatial and temporal waveform dictionaries for low-data regimes](https://mlsanity.com/paper/2607.22508/) (2026)
- [S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring](https://mlsanity.com/paper/2607.27913/) (2026)
- [MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation](https://mlsanity.com/paper/2608.29304/) (2026)
- [BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification](https://mlsanity.com/paper/2609.12218/) (2026)
- [Representational and Functional Robustness to Electrode Montages in EEG Foundation Models](https://mlsanity.com/paper/2609.36288/) (2026)
- [NeurDuo-EEG: A Long-Sequence EEG Foundation Model with Persistent State and Explicit Memory](https://mlsanity.com/paper/2609.38587/) (2026)

## Abstract (arXiv)

Biological signals, such as electroencephalograms (EEG), play a crucial role in numerous clinical applications, exhibiting diverse data formats and quality profiles. Current deep learning models for biosignals are typically specialized for specific datasets and clinical settings, limiting their broader applicability. Motivated by the success of large language models in text processing, we explore the development of foundational models that are trained from multiple data sources and can be fine-tuned on different downstream biosignal tasks. To overcome the unique challenges associated with biosignals of various formats, such as mismatched channels, variable sample lengths, and prevalent missing values, we propose a Biosignal Transformer (method). The proposed method model can enable cross-data learning with mismatched channels, variable lengths, and missing values by tokenizing diverse biosignals into unified "biosignal sentences". Specifically, we tokenize each channel into fixed-length segments containing local signal features, flattening them to form consistent "sentences". Channel embeddings and em relative position embeddings are added to preserve spatio-temporal features. The method model is versatile and applicable to various biosignal learning settings across different datasets, including joint pre-training for larger models. Comprehensive evaluations on EEG, electrocardiogram (ECG), and human activity sensory signals demonstrate that method outperforms robust baselines in common settings and facilitates learning across multiple datasets with different formats. Use CHB-MIT seizure detection task as an example, our vanilla method model shows 3% improvement over baselines in balanced accuracy, and the pre-trained method models (optimized from other data sources) can further bring up to 4% improvements.