# MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training

arXiv: https://arxiv.org/abs/2602.02494 · HTML: https://mlsanity.com/paper/2602.02494/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [Temporally Consistent Transformers for Video Generation](https://mlsanity.com/paper/2210.02396/) (2022)
- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals](https://mlsanity.com/paper/2505.18185/) (2025)
- [The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning](https://mlsanity.com/paper/2406.04328/) (2024)
- [LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale](https://mlsanity.com/paper/2506.02098/) (2025)
- [The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset](https://mlsanity.com/paper/2506.10165/) (2025)
- [Unlocking Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2505.13446/) (2025)
- [Are Large Brainwave Foundation Models Capable Yet? Insights from Fine-tuning](https://mlsanity.com/paper/2507.01196/) (2025)
- [Neural Codecs as Biosignal Tokenizers](https://mlsanity.com/paper/2510.09095/) (2025)
- [Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction](https://mlsanity.com/paper/2501.04844/) (2025)

## Built on by (most cited first, 7 of 7)

- [LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale](https://mlsanity.com/paper/2608.25204/) (2026)
- [A Common Measure of Communication for Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.02887/) (2026)
- [The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100](https://mlsanity.com/paper/2609.03231/) (2026)
- [HDND: Hierarchical Dynamic Neural Decoding for Multilingual Word/Character Retrieval from Non-Invasive Brain Recordings](https://mlsanity.com/paper/2609.24095/) (2026)
- [Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond](https://mlsanity.com/paper/2609.27650/) (2026)
- [A Roadmap for MEG Foundation Models](https://mlsanity.com/paper/2609.04461/) (2026)
- [Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2609.40359/) (2026)

## Abstract (arXiv)

Clinical brain-to-text interfaces are designed for paralysed patients who cannot provide extensive training recordings. Pre-training improves data-efficient generalisation by learning statistical priors across subjects, but these priors critically depend on context. While natural speech might unfold gradually over minutes, most methods pre-train with only a few seconds of context. Thus, we propose MEG-XL, a model pre-trained with 2.5 minutes of MEG context per sample, 5-300x longer than prior work, and equivalent to 191k tokens, capturing extended neural context. Fine-tuning on the task of word decoding from brain data, MEG-XL matches supervised performance with a fraction of the data (e.g. 1hr vs 50hrs) and outperforms brain foundation models. We find that models pre-trained with longer contexts learn representations that transfer better to word decoding. Our results indicate that long-context pre-training helps exploit extended neural context that other methods unnecessarily discard. Code, model weights, and instructions are available at https://github.com/neural-processing-lab/MEG-XL .