mlsanityData to 14 Oct 2026

Mechanistically Interpretable Neural Encoding Reveals Fine-Grained Functional Selectivity in Human Visual Cortex

arXiv 2605.16468 ↗First posted 15 May 2026cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A central goal in understanding human vision is to uncover the visual features that drive neuronal activity. A growing body of work has used artificial neural networks as encoding models to predict cortical responses to natural images, revealing the visual content that activates category-selective regions. However, existing approaches are largely correlational and treat the encoder as a black box, leaving open which image features drive each voxel's response. We introduce Mechanistically Interpretable Neural Encoding (MINE), a framework that opens this black box by applying mechanistic-interpretability tools to localize the features within natural images that drive millimeter-scale (voxel-level) activity. MINE predicts each voxel's response using language-aligned image representations, and produces semantically interpretable descriptions of the features critical for the voxel's activation. We further generalize these per-image features into per-voxel functional profiles. To validate the per-image descriptions, we show they are sufficient to generate images that elicit voxel responses matching the responses to the original images, more accurately than images generated from random or low-attribution controls. Moreover, counterfactually inserting or removing the predicted features from images shifts activation in the expected direction, providing causal evidence. Counterfactual editing guided by the per-voxel activation profiles produces even stronger activation shifts, indicating that the profiles faithfully capture each voxel's selectivity. Finally, we apply MINE to well-studied category-selective brain regions, showing it recovers their known categorical preferences while revealing fine-grained unique voxel structure within each region. Overall, our results establish mechanistic interpretability as a path to discover and causally validate fine-grained hypotheses about neural function.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 75 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

46 of 75 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
34 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  3. 2019Attention is not Explanationcited by 929
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  11. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  12. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  13. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  14. 2025Why do LLMs attend to the first token?cited by 127
  15. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  16. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  17. 2021An Interpretability Illusion for BERTcited by 97
  18. 2023Explaining black box text modules in natural language with language modelscited by 62
  19. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  20. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 41
  21. 2023Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Modelscited by 29
  22. 2023BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivitycited by 19
  23. 2024The Wisdom of a Crowd of Brains: A Universal Brain Encodercited by 15
  24. 2024The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithmscited by 13
  25. 2024Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformerscited by 13
  26. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  27. 2025Transformer brain encoders explain human high-level visual responsescited by 10
  28. 2023Causal Analysis for Robust Interpretability of Neural Networkscited by 8
  29. 2021More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activitycited by 5
  30. 2025BrainExplore: Large-Scale Discovery of Interpretable Visual Representations in the Human Braincited by 5
  31. 2025LaVCa: LLM-assisted Visual Cortex Captioningcited by 4
  32. 2025Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortexcited by 4
  33. 2024Vision-Language Models Align with Human Neural Representations in Concept Processingcited by 2
  34. 2025Towards Interpretable Visual Decoding with Attention to Brain Representationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
  2. 2025Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding
  3. 2023Reconstructing seen images from human brain activity via guided stochastic search
  4. 2026Representation as a Bottleneck for Mechanistic Interpretability: The Manifestation Unit Protocol
  5. 2023Identifying Interpretable Visual Features in Artificial and Biological Neural Systems
  6. 2025Sparse Autoencoders Bridge The Deep Learning Model and The Brain
  7. 2023Scale Alone Does not Improve Mechanistic Interpretability in Vision Models
  8. 2025NeuroVolve: Evolving Visual Stimuli toward Programmable Neural Objectives