# Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning

arXiv: https://arxiv.org/abs/2406.16910 · HTML: https://mlsanity.com/paper/2406.16910/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Deep Learning Human Mind for Automated Visual Classification](https://mlsanity.com/paper/1609.00344/) (2016)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [MAtt: A Manifold Attention Network for EEG Decoding](https://mlsanity.com/paper/2210.01986/) (2022)
- [Learning Robust Deep Visual Representations from EEG Brain Recordings](https://mlsanity.com/paper/2310.16532/) (2023)
- [Object classification from randomized EEG trials](https://mlsanity.com/paper/2004.06046/) (2020)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity](https://mlsanity.com/paper/2208.03666/) (2022)
- [SSVEP-DAN: A Data Alignment Network for SSVEP-based Brain Computer Interfaces](https://mlsanity.com/paper/2311.12666/) (2023)

## Built on by (most cited first, 12 of 12)

- [Quantum Adaptive Self-Attention for Quantum Transformer Models](https://mlsanity.com/paper/2504.05336/) (2025)
- [NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models](https://mlsanity.com/paper/2410.00712/) (2024)
- [Large Cognition Model: Towards Pretrained EEG Foundation Model](https://mlsanity.com/paper/2502.17464/) (2025)
- [EEG2TEXT-CN: An Exploratory Study of Open-Vocabulary Chinese Text-EEG Alignment via Large Language Model and Contrastive Learning on ChineseEEG](https://mlsanity.com/paper/2506.00854/) (2025)
- [Quantum-Enhanced Natural Language Generation: A Multi-Model Framework with Hybrid Quantum-Classical Architectures](https://mlsanity.com/paper/2508.21332/) (2025)
- [Exploring the Potential of QEEGNet for Cross-Task and Cross-Dataset Electroencephalography Encoding with Quantum Machine Learning](https://mlsanity.com/paper/2503.00080/) (2025)
- [SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval](https://mlsanity.com/paper/2603.20738/) (2026)
- [Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing](https://mlsanity.com/paper/2503.10663/) (2025)
- [Optimizing Supply Chain Networks with the Power of Graph Neural Networks](https://mlsanity.com/paper/2501.06221/) (2025)
- [Decoding Visual Neural Representations by Multimodal with Dynamic Balancing](https://mlsanity.com/paper/2509.03433/) (2025)
- [A Unified SPD Token Transformer Framework for EEG Classification: Systematic Comparison of Geometric Embeddings](https://mlsanity.com/paper/2601.21521/) (2026)
- [Toward High-Fidelity Visual Reconstruction: From EEG-Based Conditioned Generation to Joint-Modal Guided Rebuilding](https://mlsanity.com/paper/2603.19667/) (2026)

## Abstract (arXiv)

Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG.