# Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

arXiv: https://arxiv.org/abs/2112.02690 · HTML: https://mlsanity.com/paper/2112.02690/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Does the brain represent words? An evaluation of brain decoding studies of language understanding](https://mlsanity.com/paper/1806.00591/) (2018)
- [Human brain activity for machine attention](https://mlsanity.com/paper/2006.05113/) (2020)
- [Understanding language-elicited EEG data by predicting it from a fine-tuned language model](https://mlsanity.com/paper/1904.01548/) (2019)

## Built on by (most cited first, 30 of 67)

- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [Understanding Emergent In-Context Learning from a Kernel Regression Perspective](https://mlsanity.com/paper/2305.12766/) (2023)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language](https://mlsanity.com/paper/2307.05355/) (2023)
- [Through their eyes: multi-subject Brain Decoding with simple alignment techniques](https://mlsanity.com/paper/2309.00627/) (2023)
- [The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning](https://mlsanity.com/paper/2406.04328/) (2024)
- [EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer](https://mlsanity.com/paper/2405.02165/) (2024)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder](https://mlsanity.com/paper/2402.17433/) (2024)
- [MAD: Multi-Alignment MEG-to-Text Decoding](https://mlsanity.com/paper/2406.01512/) (2024)
- [Unlocking Non-Invasive Brain-to-Text](https://mlsanity.com/paper/2505.13446/) (2025)
- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://mlsanity.com/paper/2208.06348/) (2022)
- [BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification](https://mlsanity.com/paper/2406.06786/) (2024)
- [NeuSpeech: Decode Neural signal as Speech](https://mlsanity.com/paper/2403.01748/) (2024)
- [Large Transformers are Better EEG Learners](https://mlsanity.com/paper/2308.11654/) (2023)
- [Unveiling Thoughts: A Review of Advancements in EEG Brain Signal Decoding into Text](https://mlsanity.com/paper/2405.00726/) (2024)
- [BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding](https://mlsanity.com/paper/2409.00121/) (2024)
- [Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding](https://mlsanity.com/paper/2312.09430/) (2023)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [NeuGPT: Unified multi-modal Neural GPT](https://mlsanity.com/paper/2410.20916/) (2024)
- [SEE: Semantically Aligned EEG-to-Text Translation](https://mlsanity.com/paper/2409.16312/) (2024)
- [Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction](https://mlsanity.com/paper/2501.04844/) (2025)
- [WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities](https://mlsanity.com/paper/2510.00032/) (2025)
- [Semantic-aware Contrastive Learning for Electroencephalography-to-Text Generation with Curriculum Learning](https://mlsanity.com/paper/2301.09237/) (2023)
- [BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision](https://mlsanity.com/paper/2309.12056/) (2023)
- [Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation](https://mlsanity.com/paper/2505.17099/) (2025)
- [Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies](https://mlsanity.com/paper/2510.22095/) (2025)
- [HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment](https://mlsanity.com/paper/2603.22721/) (2026)
- [Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings](https://mlsanity.com/paper/2408.04679/) (2024)

## Abstract (arXiv)

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. In addition, most of the high-performing approaches require data from invasive devices (e.g., ECoG). In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. We hypothesis that the human brain functions as a special text encoder and propose a novel framework leveraging pre-trained language models (e.g., BART). Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for a high-performance open vocabulary brain-to-text system once sufficient data is available