# Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

arXiv: https://arxiv.org/abs/2102.08655 · HTML: https://mlsanity.com/paper/2102.08655/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 117 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Comparative Study of CNN and RNN for Natural Language Processing](https://mlsanity.com/paper/1702.01923/) (2017)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [Inducing brain-relevant bias in natural language processing models](https://mlsanity.com/paper/1911.03268/) (2019)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Improved Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1708.01204/) (2017)
- [Entity Recognition at First Sight: Improving NER with Eye Movement Information](https://mlsanity.com/paper/1902.10068/) (2019)
- [Leveraging Cognitive Features for Sentiment Analysis](https://mlsanity.com/paper/1701.05581/) (2017)
- [Advancing NLP with Cognitive Language Processing Signals](https://mlsanity.com/paper/1904.02682/) (2019)
- [Contextual Embeddings: When Are They Worth It?](https://mlsanity.com/paper/2005.09117/) (2020)
- [Brain2Word: Decoding Brain Activity for Language Generation](https://mlsanity.com/paper/2009.04765/) (2020)
- [Sentiment analysis is not solved! Assessing and probing sentiment classification](https://mlsanity.com/paper/1906.05887/) (2019)
- [Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets](https://mlsanity.com/paper/1709.04219/) (2017)
- [Does the brain represent words? An evaluation of brain decoding studies of language understanding](https://mlsanity.com/paper/1806.00591/) (2018)
- [Robust Evaluation of Language-Brain Encoding Experiments](https://mlsanity.com/paper/1904.02547/) (2019)

## Built on by (most cited first, 13 of 13)

- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)
- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://mlsanity.com/paper/2208.06348/) (2022)
- [BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification](https://mlsanity.com/paper/2406.06786/) (2024)
- [BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision](https://mlsanity.com/paper/2309.12056/) (2023)
- [The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models](https://mlsanity.com/paper/2310.07106/) (2023)
- [EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More](https://mlsanity.com/paper/2407.01884/) (2024)
- [Text Difficulty Study: Do machines behave the same as humans regarding text difficulty?](https://mlsanity.com/paper/2208.14509/) (2022)
- [Neural Memory Decoding with EEG Data and Representation Learning](https://mlsanity.com/paper/2307.13181/) (2023)
- [Large-scale Foundation Models and Generative AI for BigData Neuroscience](https://mlsanity.com/paper/2310.18377/) (2023)
- [Feature Estimation of Global Language Processing in EEG Using Attention Maps](https://mlsanity.com/paper/2409.19174/) (2024)
- [Detecting Reading-Induced Confusion Using EEG and Eye Tracking](https://mlsanity.com/paper/2508.14442/) (2025)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)

## Abstract (arXiv)

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity to this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, further research is needed. Finally, EEG data shows to be particularly promising when limited training data is available.