# NeuSpeech: Decode Neural signal as Speech

arXiv: https://arxiv.org/abs/2403.01748 · HTML: https://mlsanity.com/paper/2403.01748/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning](https://mlsanity.com/paper/2305.15212/) (2023)
- [UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language](https://mlsanity.com/paper/2307.05355/) (2023)

## Built on by (most cited first, 11 of 11)

- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale](https://mlsanity.com/paper/2506.02098/) (2025)
- [The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset](https://mlsanity.com/paper/2506.10165/) (2025)
- [Unlocking Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2505.13446/) (2025)
- [Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset](https://mlsanity.com/paper/2510.21038/) (2025)
- [The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding](https://mlsanity.com/paper/2609.10296/) (2026)
- [MEGState: Phoneme Decoding from Magnetoencephalography Signals](https://mlsanity.com/paper/2512.17978/) (2025)
- [Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond](https://mlsanity.com/paper/2609.27650/) (2026)
- [MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors](https://mlsanity.com/paper/2605.31173/) (2026)
- [A Roadmap for MEG Foundation Models](https://mlsanity.com/paper/2609.04461/) (2026)
- [BAT-CLIP: Trimodal Alignment of Brain, Audio and Text](https://mlsanity.com/paper/2609.31180/) (2026)

## Abstract (arXiv)

Decoding language from brain dynamics is an important open direction in the realm of brain-computer interface (BCI), especially considering the rapid growth of large language models. Compared to invasive-based signals which require electrode implantation surgery, non-invasive neural signals (e.g. EEG, MEG) have attracted increasing attention considering their safety and generality. However, the exploration is not adequate in three aspects: 1) previous methods mainly focus on EEG but none of the previous works address this problem on MEG with better signal quality; 2) prior works have predominantly used ``teacher-forcing" during generative decoding, which is impractical; 3) prior works are mostly ``BART-based" not fully auto-regressive, which performs better in other sequence tasks. In this paper, we explore the brain-to-text translation of MEG signals in a speech-decoding formation. Here we are the first to investigate a cross-attention-based ``whisper" model for generating text directly from MEG signals without teacher forcing. Our model achieves impressive BLEU-1 scores of 60.30 and 52.89 without pretraining & teacher-forcing on two major datasets (textitGWilliams and textitSchoffelen). This paper conducts a comprehensive review to understand how speech decoding formation performs on the neural decoding tasks, including pretraining initialization, training & evaluation set splitting, augmentation, and scaling law. Code is available at https://github.com/NeuSpeech/NeuSpeech1.