# EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More

arXiv: https://arxiv.org/abs/2407.01884 · HTML: https://mlsanity.com/paper/2407.01884/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More](https://mlsanity.com/paper/2403.12534/) (2024)
- [EEG2IMAGE: Image Reconstruction from EEG Brain Signals](https://mlsanity.com/paper/2302.10121/) (2023)
- [Learning Robust Deep Visual Representations from EEG Brain Recordings](https://mlsanity.com/paper/2310.16532/) (2023)
- [EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2403.14082/) (2024)
- [Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera](https://mlsanity.com/paper/2309.09297/) (2023)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [Object classification from randomized EEG trials](https://mlsanity.com/paper/2004.06046/) (2020)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2310.02296/) (2023)
- [Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation](https://mlsanity.com/paper/2401.17664/) (2024)
- [MindBigData 2022 A Large Dataset of Brain Signals](https://mlsanity.com/paper/2212.14746/) (2022)

## Built on by (most cited first, 2 of 2)

- [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://mlsanity.com/paper/2410.07507/) (2024)
- [Category-aware EEG image generation based on wavelet transform and contrast semantic loss](https://mlsanity.com/paper/2505.24301/) (2025)

## Abstract (arXiv)

Recently, electroencephalography (EEG) signals have been actively incorporated to decode brain activity to visual or textual stimuli and achieve object recognition in multi-modal AI. Accordingly, endeavors have been focused on building EEG-based datasets from visual or textual single-modal stimuli. However, these datasets offer limited EEG epochs per category, and the complex semantics of stimuli presented to participants compromise their quality and fidelity in capturing precise brain activity. The study in neuroscience unveils that the relationship between visual and textual stimulus in EEG recordings provides valuable insights into the brain's ability to process and integrate multi-modal information simultaneously. Inspired by this, we propose a novel large-scale multi-modal dataset, named EIT-1M, with over 1 million EEG-image-text pairs. Our dataset is superior in its capacity of reflecting brain activities in simultaneously processing multi-modal information. To achieve this, we collected data pairs while participants viewed alternating sequences of visual-textual stimuli from 60K natural images and category-specific texts. Common semantic categories are also included to elicit better reactions from participants' brains. Meanwhile, response-based stimulus timing and repetition across blocks and sessions are included to ensure data diversity. To verify the effectiveness of EIT-1M, we provide an in-depth analysis of EEG data captured from multi-modal stimuli across different categories and participants, along with data quality scores for transparency. We demonstrate its validity on two tasks: 1) EEG recognition from visual or textual stimuli or both and 2) EEG-to-visual generation.