# NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models arXiv: https://arxiv.org/abs/2410.00712 ยท HTML: https://mlsanity.com/paper/2410.00712/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. - Builds on 27 of its 57 references that are in the map. - First posted on arXiv 1 Oct 2024. ## Placed in - Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/) ## Builds on (most cited first, 27 of 27) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016) - [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023) - [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019) - [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023) - [LCM-LoRA: A Universal Stable-Diffusion Acceleration Module](https://mlsanity.com/paper/2311.05556/) (2023) - [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023) - [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023) - [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024) - [Deep Learning Human Mind for Automated Visual Classification](https://mlsanity.com/paper/1609.00344/) (2016) - [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023) - [Generative Adversarial Networks](https://mlsanity.com/paper/1406.2661/) (2014) - [EEG2IMAGE: Image Reconstruction from EEG Brain Signals](https://mlsanity.com/paper/2302.10121/) (2023) - [BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction](https://mlsanity.com/paper/2312.14871/) (2023) - [Visual Neural Decoding via Improved Visual-EEG Semantic Consistency](https://mlsanity.com/paper/2408.06788/) (2024) - [Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals](https://mlsanity.com/paper/2308.02510/) (2023) - [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023) - [QEEGNet: Quantum Machine Learning for Enhanced Electroencephalography Encoding](https://mlsanity.com/paper/2407.19214/) (2024) - [Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning](https://mlsanity.com/paper/2406.16910/) (2024) - [Quantum Multimodal Contrastive Learning Framework](https://mlsanity.com/paper/2408.13919/) (2024) ## Built on by (most cited first, 10 of 10) - [Quantum Adaptive Self-Attention for Quantum Transformer Models](https://mlsanity.com/paper/2504.05336/) (2025) - [Large Cognition Model: Towards Pretrained EEG Foundation Model](https://mlsanity.com/paper/2502.17464/) (2025) - [EEG2TEXT-CN: An Exploratory Study of Open-Vocabulary Chinese Text-EEG Alignment via Large Language Model and Contrastive Learning on ChineseEEG](https://mlsanity.com/paper/2506.00854/) (2025) - [Quantum-Enhanced Natural Language Generation: A Multi-Model Framework with Hybrid Quantum-Classical Architectures](https://mlsanity.com/paper/2508.21332/) (2025) - [Exploring the Potential of QEEGNet for Cross-Task and Cross-Dataset Electroencephalography Encoding with Quantum Machine Learning](https://mlsanity.com/paper/2503.00080/) (2025) - [Category-aware EEG image generation based on wavelet transform and contrast semantic loss](https://mlsanity.com/paper/2505.24301/) (2025) - [SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval](https://mlsanity.com/paper/2603.20738/) (2026) - [Vision-QRWKV: Exploring Quantum-Enhanced RWKV Models for Image Classification](https://mlsanity.com/paper/2506.06633/) (2025) - [A Unified SPD Token Transformer Framework for EEG Classification: Systematic Comparison of Geometric Embeddings](https://mlsanity.com/paper/2601.21521/) (2026) - [Toward High-Fidelity Visual Reconstruction: From EEG-Based Conditioned Generation to Joint-Modal Guided Rebuilding](https://mlsanity.com/paper/2603.19667/) (2026) ## Abstract (arXiv) NECOMIMI (NEural-COgnitive MultImodal EEG-Informed Image Generation with Diffusion Models) introduces a novel framework for generating images directly from EEG signals using advanced diffusion models. Unlike previous works that focused solely on EEG-image classification through contrastive learning, NECOMIMI extends this task to image generation. The proposed NERV EEG encoder demonstrates state-of-the-art (SoTA) performance across multiple zero-shot classification tasks, including 2-way, 4-way, and 200-way, and achieves top results in our newly proposed Category-based Assessment Table (CAT) Score, which evaluates the quality of EEG-generated images based on semantic concepts. A key discovery of this work is that the model tends to generate abstract or generalized images, such as landscapes, rather than specific objects, highlighting the inherent challenges of translating noisy and low-resolution EEG data into detailed visual outputs. Additionally, we introduce the CAT Score as a new metric tailored for EEG-to-image evaluation and establish a benchmark on the ThingsEEG dataset. This study underscores the potential of EEG-to-image generation while revealing the complexities and challenges that remain in bridging neural activity with visual representation.