# DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation

arXiv: https://arxiv.org/abs/2309.14030 · HTML: https://mlsanity.com/paper/2309.14030/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2023.

## Built on by (most cited first, 30 of 36)

- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)
- [EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer](https://mlsanity.com/paper/2405.02165/) (2024)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals](https://mlsanity.com/paper/2405.11459/) (2024)
- [Unveiling Thoughts: A Review of Advancements in EEG Brain Signal Decoding into Text](https://mlsanity.com/paper/2405.00726/) (2024)
- [BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding](https://mlsanity.com/paper/2409.00121/) (2024)
- [Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding](https://mlsanity.com/paper/2312.09430/) (2023)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities](https://mlsanity.com/paper/2510.00032/) (2025)
- [Large Language Models for EEG: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2506.06353/) (2025)
- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding](https://mlsanity.com/paper/2502.17465/) (2025)
- [ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification](https://mlsanity.com/paper/2506.14783/) (2025)
- [Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset](https://mlsanity.com/paper/2510.21038/) (2025)
- [Enhancing Traffic Prediction with Textual Data Using Large Language Models](https://mlsanity.com/paper/2405.06719/) (2024)
- [LiPCoT: Linear Predictive Coding based Tokenizer for Self-supervised Learning of Time Series Data via Language Models](https://mlsanity.com/paper/2408.07292/) (2024)
- [Non-invasive Neural Decoding in Source Reconstructed Brain Space](https://mlsanity.com/paper/2410.19838/) (2024)
- [Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing](https://mlsanity.com/paper/2505.10356/) (2025)
- [TSVer: A Benchmark for Fact Verification Against Time-Series Evidence](https://mlsanity.com/paper/2511.01101/) (2025)
- [SI-SD: Sleep Interpreter through awake-guided cross-subject Semantic Decoding](https://mlsanity.com/paper/2309.16457/) (2023)
- [The VEP Booster: A Closed-Loop AI System for Visual EEG Biomarker Auto-generation](https://mlsanity.com/paper/2407.15167/) (2024)
- [Bridging Brain with Foundation Models through Self-Supervised Learning](https://mlsanity.com/paper/2506.16009/) (2025)
- [Decoding Visual Neural Representations by Multimodal with Dynamic Balancing](https://mlsanity.com/paper/2509.03433/) (2025)
- [Neurocognitive Modeling for Text Generation: Deep Learning Architecture for EEG Data](https://mlsanity.com/paper/2509.07202/) (2025)
- [DELTA: Language Diffusion-based EEG-to-Text Architecture](https://mlsanity.com/paper/2511.21746/) (2025)
- [Prompting Underestimates LLM Capability for Time Series Classification](https://mlsanity.com/paper/2601.03464/) (2026)
- [Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes](https://mlsanity.com/paper/2602.01247/) (2026)

## Abstract (arXiv)

The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs). With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it realizes translation on raw waves without marker by introducing text-EEG contrastive alignment training, and 2) it alleviates the interference caused by individual differences in EEG waves through an invariant discrete codex with or without markers. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. This work is the first to facilitate the translation of entire EEG signal periods without word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset.