# Brain2Word: Decoding Brain Activity for Language Generation

arXiv: https://arxiv.org/abs/2009.04765 · HTML: https://mlsanity.com/paper/2009.04765/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2020.

## Built on by (most cited first, 15 of 15)

- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language](https://mlsanity.com/paper/2307.05355/) (2023)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Unveiling Thoughts: A Review of Advancements in EEG Brain Signal Decoding into Text](https://mlsanity.com/paper/2405.00726/) (2024)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [BrainLLM: Generative Language Decoding from Brain Recordings](https://mlsanity.com/paper/2311.09889/) (2023)
- [Structural Similarities Between Language Models and Neural Response Measurements](https://mlsanity.com/paper/2306.01930/) (2023)
- [BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2502.15172/) (2025)
- [How Many Bytes Can You Take Out Of Brain-To-Text Decoding?](https://mlsanity.com/paper/2405.14055/) (2024)
- [Cross-view Brain Decoding](https://mlsanity.com/paper/2204.09564/) (2022)
- [Language Reconstruction with Brain Predictive Coding from fMRI Data](https://mlsanity.com/paper/2405.11597/) (2024)
- [Decoding Linguistic Representations of Human Brain](https://mlsanity.com/paper/2407.20622/) (2024)
- [Improve Language Model and Brain Alignment via Associative Memory](https://mlsanity.com/paper/2505.13844/) (2025)

## Abstract (arXiv)

Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of language stimuli, recent studies have shown that it is possible to decode fMRI scans into an embedding of the word a subject is reading. However, such word embeddings are designed for natural language processing tasks rather than for brain decoding. Therefore, they limit our ability to recover the precise stimulus. In this work, we propose to directly classify an fMRI scan, mapping it to the corresponding word within a fixed vocabulary. Unlike existing work, we evaluate on scans from previously unseen subjects. We argue that this is a more realistic setup and we present a model that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1 and 13.59% Top-5 accuracy in this challenging task, significantly outperforming all the considered competitive baselines. Furthermore, we use the decoded words to guide language generation with the GPT-2 model. This way, we advance the quest for a system that translates brain activities into coherent text.