# Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)

arXiv: https://arxiv.org/abs/2410.07507 · HTML: https://mlsanity.com/paper/2410.07507/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer](https://mlsanity.com/paper/2405.02165/) (2024)
- [MindBigData 2022 A Large Dataset of Brain Signals](https://mlsanity.com/paper/2212.14746/) (2022)
- [EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More](https://mlsanity.com/paper/2407.01884/) (2024)

## Built on by (most cited first, 18 of 18)

- [A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation](https://mlsanity.com/paper/2411.12157/) (2024)
- [LLM as Clinical Graph Structure Refiner: Enhancing Representation Learning in EEG Seizure Diagnosis](https://mlsanity.com/paper/2604.28178/) (2026)
- [WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities](https://mlsanity.com/paper/2510.00032/) (2025)
- [Large Language Models for EEG: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2506.06353/) (2025)
- [Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies](https://mlsanity.com/paper/2510.22095/) (2025)
- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval](https://mlsanity.com/paper/2603.17109/) (2026)
- [DELTA: Language Diffusion-based EEG-to-Text Architecture](https://mlsanity.com/paper/2511.21746/) (2025)
- [Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding](https://mlsanity.com/paper/2603.03312/) (2026)
- [RAG-based EEG-to-Text Translation Using Deep Learning and LLMs](https://mlsanity.com/paper/2605.17503/) (2026)
- [Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence](https://mlsanity.com/paper/2608.16975/) (2026)
- [Decoding Open-Ended Information Seeking Goals from Eye Movements in Reading](https://mlsanity.com/paper/2505.02872/) (2025)
- [Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions](https://mlsanity.com/paper/2603.12296/) (2026)
- [The Deep-Match Framework for Event-Related Potential Detection in EEG](https://mlsanity.com/paper/2603.20258/) (2026)
- [MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data](https://mlsanity.com/paper/2606.20696/) (2026)
- [Large language models in medical time series analysis](https://mlsanity.com/paper/2609.22262/) (2026)
- [BAT-CLIP: Trimodal Alignment of Brain, Audio and Text](https://mlsanity.com/paper/2609.31180/) (2026)

## Abstract (arXiv)

Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli and text captions demonstrate the efficacy of multimodal LLMs (LLaMA-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, as well as fluency and adequacy measures. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing.