# EEG-EditBench: Probing Visual Information in EEG-Image Retrieval Models with Controlled Image Edits

arXiv: https://arxiv.org/abs/2607.27857 · HTML: https://mlsanity.com/paper/2607.27857/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)
- [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2404.09990/) (2024)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing](https://mlsanity.com/paper/2408.14180/) (2024)
- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information](https://mlsanity.com/paper/2412.10489/) (2024)
- [Learning Robust Deep Visual Representations from EEG Brain Recordings](https://mlsanity.com/paper/2310.16532/) (2023)
- [NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment](https://mlsanity.com/paper/2511.06836/) (2025)
- [NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery](https://mlsanity.com/paper/2506.06898/) (2025)
- [Shrinking the Teacher: An Adaptive Teaching Paradigm for Asymmetric EEG-Vision Alignment](https://mlsanity.com/paper/2511.11422/) (2025)
- [HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment](https://mlsanity.com/paper/2603.22721/) (2026)
- [Exploring The Visual Feature Space for Multimodal Neural Decoding](https://mlsanity.com/paper/2505.15755/) (2025)
- [Multigranular Evaluation for Brain Visual Decoding](https://mlsanity.com/paper/2507.07993/) (2025)
- [CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment](https://mlsanity.com/paper/2603.12722/) (2026)

## Abstract (arXiv)

Recent EEG-to-image retrieval models have achieved strong performance in identifying viewed images from semantically diverse candidates. Yet such success does not reveal what visual information supports the match. A model may readily identify a cheetah among tools, plants, and vehicles, but can it still distinguish the viewed cheetah from the same scene with the cheetah replaced by a dog? Motivated by this question, we introduce EEG-EditBench, a diagnostic benchmark that examines this question through controlled edits of object identity, attributes, background, and object presence. Built from the 200 THINGS-EEG2 test images, EEG-EditBench contains 2,137 quality-controlled edits and evaluates eight representative EEG visual decoding models. Our results show that strong standard retrieval does not consistently transfer to edit-based evaluation, with fine-grained attribute changes presenting the greatest challenge. EEG-EditBench reveals model behavior hidden by aggregate retrieval accuracy and provides a controlled basis for studying what visual information EEG-image models preserve. The code and complete dataset are publicly available.