# Decoding Linguistic Representations of Human Brain

arXiv: https://arxiv.org/abs/2407.20622 · HTML: https://mlsanity.com/paper/2407.20622/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 166 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [DualGAN: Unsupervised Dual Learning for Image-to-Image Translation](https://mlsanity.com/paper/1704.02510/) (2017)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021)
- [VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior](https://mlsanity.com/paper/2312.01841/) (2023)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains](https://mlsanity.com/paper/1906.01539/) (2019)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language](https://mlsanity.com/paper/2307.05355/) (2023)
- [Brain2Word: Decoding Brain Activity for Language Generation](https://mlsanity.com/paper/2009.04765/) (2020)
- [MAD: Multi-Alignment MEG-to-Text Decoding](https://mlsanity.com/paper/2406.01512/) (2024)
- [Relating Simple Sentence Representations in Deep Neural Networks and the Brain](https://mlsanity.com/paper/1906.11861/) (2019)
- [Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain](https://mlsanity.com/paper/2303.14871/) (2023)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)

## Built on by (most cited first, 1 of 1)

- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)

## Abstract (arXiv)

Language, as an information medium created by advanced organisms, has always been a concern of neuroscience regarding how it is represented in the brain. Decoding linguistic representations in the evoked brain has shown groundbreaking achievements, thanks to the rapid improvement of neuroimaging, medical technology, life sciences and artificial intelligence. In this work, we present a taxonomy of brain-to-language decoding of both textual and speech formats. This work integrates two types of research: neuroscience focusing on language understanding and deep learning-based brain decoding. Generating discernible language information from brain activity could not only help those with limited articulation, especially amyotrophic lateral sclerosis (ALS) patients but also open up a new way for the next generation's brain-computer interface (BCI). This article will help brain scientists and deep-learning researchers to gain a bird's eye view of fine-grained language perception, and thus facilitate their further investigation and research of neural process and language decoding.