# The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning

arXiv: https://arxiv.org/abs/2406.04328 · HTML: https://mlsanity.com/paper/2406.04328/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 76 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model](https://mlsanity.com/paper/2401.10278/) (2024)
- [SEANet: A Multi-modal Speech Enhancement Network](https://mlsanity.com/paper/2009.02095/) (2020)
- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signals](https://mlsanity.com/paper/2306.13102/) (2023)
- [BrainWave: A Brain Signal Foundation Model for Clinical Applications](https://mlsanity.com/paper/2402.10251/) (2024)
- [Self-supervised representation learning from electroencephalography signals](https://mlsanity.com/paper/1911.05419/) (2019)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [Group-level Brain Decoding with Deep Learning](https://mlsanity.com/paper/2205.14102/) (2022)
- [Semantic reconstruction of continuous language from MEG signals](https://mlsanity.com/paper/2309.07701/) (2023)
- [A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation](https://mlsanity.com/paper/2304.05369/) (2023)

## Built on by (most cited first, 21 of 21)

- [LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale](https://mlsanity.com/paper/2506.02098/) (2025)
- [The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset](https://mlsanity.com/paper/2506.10165/) (2025)
- [Unlocking Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2505.13446/) (2025)
- [MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training](https://mlsanity.com/paper/2602.02494/) (2026)
- [Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordings](https://mlsanity.com/paper/2410.19986/) (2024)
- [MEG-GPT: A transformer-based foundation model for magnetoencephalography data](https://mlsanity.com/paper/2510.18080/) (2025)
- [LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale](https://mlsanity.com/paper/2608.25204/) (2026)
- [Pretraining Large Brain Language Model for Active BCI: Silent Speech](https://mlsanity.com/paper/2504.21214/) (2025)
- [Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models](https://mlsanity.com/paper/2510.21520/) (2025)
- [Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding](https://mlsanity.com/paper/2502.17465/) (2025)
- [Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset](https://mlsanity.com/paper/2510.21038/) (2025)
- [What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval](https://mlsanity.com/paper/2605.24524/) (2026)
- [Non-invasive Neural Decoding in Source Reconstructed Brain Space](https://mlsanity.com/paper/2410.19838/) (2024)
- [NeuroTTT: Bridging Pretraining-Downstream Task Misalignment in EEG Foundation Models via Test-Time Training](https://mlsanity.com/paper/2509.26301/) (2025)
- [The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100](https://mlsanity.com/paper/2609.03231/) (2026)
- [From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining](https://mlsanity.com/paper/2512.15830/) (2025)
- [Next-Token Prediction Learns Generalisable Representations of Sleep Physiology](https://mlsanity.com/paper/2606.09605/) (2026)
- [Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift](https://mlsanity.com/paper/2604.04129/) (2026)
- [A Roadmap for MEG Foundation Models](https://mlsanity.com/paper/2609.04461/) (2026)
- [From Neurons to Conversation: Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.36736/) (2026)
- [Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2609.40359/) (2026)

## Abstract (arXiv)

The past few years have seen remarkable progress in the decoding of speech from brain activity, primarily driven by large single-subject datasets. However, due to individual variation, such as anatomy, and differences in task design and scanning hardware, leveraging data across subjects and datasets remains challenging. In turn, the field has not benefited from the growing number of open neural data repositories to exploit large-scale deep learning. To address this, we develop neuroscience-informed self-supervised objectives, together with an architecture, for learning from heterogeneous brain recordings. Scaling to nearly 400 hours of MEG data and 900 subjects, our approach shows generalisation across participants, datasets, tasks, and even to novel subjects. It achieves improvements of 15-27% over state-of-the-art models and matches surgical decoding performance with non-invasive data. These advances unlock the potential for scaling speech decoding models beyond the current frontier.