# WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities

arXiv: https://arxiv.org/abs/2510.00032 · HTML: https://mlsanity.com/paper/2510.00032/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals](https://mlsanity.com/paper/2409.00101/) (2024)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features](https://mlsanity.com/paper/1810.10974/) (2018)
- [DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation](https://mlsanity.com/paper/2309.14030/) (2023)
- [BIOT: Cross-data Biosignal Learning in the Wild](https://mlsanity.com/paper/2305.10351/) (2023)
- [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://mlsanity.com/paper/2410.07507/) (2024)
- [Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals](https://mlsanity.com/paper/2308.02510/) (2023)
- [Deep learning with convolutional neural networks for decoding and visualization of EEG pathology](https://mlsanity.com/paper/1708.08012/) (2017)
- [EEG-ITNet: An Explainable Inception Temporal Convolutional Network for Motor Imagery Classification](https://mlsanity.com/paper/2204.06947/) (2022)
- [BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding](https://mlsanity.com/paper/2409.00121/) (2024)

## Built on by (most cited first, 6 of 6)

- [Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signal](https://mlsanity.com/paper/2602.11558/) (2026)
- [DentalGPT: Incentivizing Multimodal Complex Reasoning in Dentistry](https://mlsanity.com/paper/2512.11558/) (2025)
- [Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies](https://mlsanity.com/paper/2510.22095/) (2025)
- [Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding](https://mlsanity.com/paper/2603.03312/) (2026)
- [NeuroCanvas: VLLM-Powered Robust Seizure Detection by Reformulating Multichannel EEG as Image](https://mlsanity.com/paper/2602.04769/) (2026)
- [Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding](https://mlsanity.com/paper/2610.09355/) (2026)

## Abstract (arXiv)

Electroencephalography (EEG) interpretation using multimodal large language models (MLLMs) offers a novel approach for analyzing brain signals. However, the complex nature of brain activity introduces critical challenges: EEG signals simultaneously encode both cognitive processes and intrinsic neural states, creating a mismatch in EEG paired-data modality that hinders effective cross-modal representation learning. Through a pivot investigation, we uncover complementary relationships between these modalities. Leveraging this insight, we propose mapping EEG signals and their corresponding modalities into a unified semantic space to achieve generalized interpretation. To fully enable conversational capabilities, we further introduce WaveMind-Instruct-338k, the first cross-task EEG dataset for instruction tuning. The resulting model demonstrates robust classification accuracy while supporting flexible, open-ended conversations across four downstream tasks, thereby offering valuable insights for both neuroscience research and the development of general-purpose EEG models.