mlsanityData to 7 Oct 2026

The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning

arXiv 2406.04328 ↗First posted 6 Jun 2024cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

The past few years have seen remarkable progress in the decoding of speech from brain activity, primarily driven by large single-subject datasets. However, due to individual variation, such as anatomy, and differences in task design and scanning hardware, leveraging data across subjects and datasets remains challenging. In turn, the field has not benefited from the growing number of open neural data repositories to exploit large-scale deep learning. To address this, we develop neuroscience-informed self-supervised objectives, together with an architecture, for learning from heterogeneous brain recordings. Scaling to nearly 400 hours of MEG data and 900 subjects, our approach shows generalisation across participants, datasets, tasks, and even to novel subjects. It achieves improvements of 15-27% over state-of-the-art models and matches surgical decoding performance with non-invasive data. These advances unlock the potential for scaling speech decoding models beyond the current frontier.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 30 of its 76 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

30 of 76 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2016Colorful Image Colorizationcited by 1,113
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2022High Fidelity Neural Audio Compressioncited by 673
  12. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
18 more
  1. 2016Learning Representations for Automatic Colorizationcited by 251
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  4. 2022Decoding speech perception from non-invasive brain recordingscited by 129
  5. 2015Learning to See by Movingcited by 101
  6. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  7. 2023BrainBERT: Self-supervised representation learning for intracranial recordingscited by 70
  8. 2021Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classificationcited by 67
  9. 2024EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Modelcited by 51
  10. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 35
  11. 2024Are EEG-to-Text Models Working?cited by 32
  12. 2023MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signalscited by 26
  13. 2024BrainWave: A Brain Signal Foundation Model for Clinical Applicationscited by 26
  14. 2019Self-supervised representation learning from electroencephalography signalscited by 24
  15. 2022STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformercited by 23
  16. 2022Group-level Brain Decoding with Deep Learningcited by 7
  17. 2023Semantic reconstruction of continuous language from MEG signalscited by 5
  18. 2023A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 9.

21 papers.

Sort:

  1. 2025LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scalecited by 20
  2. 2025The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Datasetcited by 16
  3. 2025Unlocking Non-Invasive Brain-to-Textcited by 14
  4. 2026MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Trainingcited by 6
  5. 2024Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordingscited by 5
  6. 2025MEG-GPT: A transformer-based foundation model for magnetoencephalography datacited by 5
  7. 2025Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Modelscited by 4
  8. 2026LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scalecited by 4
  9. 2025Pretraining Large Brain Language Model for Active BCI: Silent Speechcited by 3
  10. 2025Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Datasetcited by 3
  11. 2025Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decodingcited by 2
  12. 2024Non-invasive Neural Decoding in Source Reconstructed Brain Spacecited by 1
9 more
  1. 2025NeuroTTT: Bridging Pretraining-Downstream Task Misalignment in EEG Foundation Models via Test-Time Trainingcited by 1
  2. 2025From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretrainingcited by 1
  3. 2026What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrievalcited by 1
  4. 2026The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100cited by 1
  5. 2026Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shiftcited by 0
  6. 2026Next-Token Prediction Learns Generalisable Representations of Sleep Physiologycited by 0
  7. 2026A Roadmap for MEG Foundation Modelscited by 0
  8. 2026From Neurons to Conversation: Speech Brain-Computer Interfacescited by 0
  9. 2026Removing Timing Shortcuts Improves Non-Invasive Brain-to-Textcited by 0