# Coarse-to-fine Hierarchical Architecture with Sequential Mamba for Brain Reconstruction

arXiv: https://arxiv.org/abs/2606.04772 · HTML: https://mlsanity.com/paper/2606.04772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Physiological signal modeling, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Physiological signal modeling](https://mlsanity.com/ssm/physiological-signal-modeling/)

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [Multimodal foundation models are better simulators of the human brain](https://mlsanity.com/paper/2208.08263/) (2022)
- [SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning](https://mlsanity.com/paper/2508.10298/) (2025)

## Abstract (arXiv)

Understanding the relationship between deep visual representations and the human visual system is a fundamental challenge in computational neuroscience. While modern vision models achieve strong performance in image recognition, their correspondence with the hierarchical organization of the human visual cortex remains an open question. In this study, we propose CHASMBrain, a novel hierarchical two-stage framework for image-to-fMRI encoding. Our architecture leverages a dual-stream Mamba design to explicitly separate and process global semantic tokens and local spatial patches, motivated by the functional organization of the visual cortex. A coarse-to-fine strategy is employed: Stage 1 predicts denoised ROI-level activations, while Stage 2 refines these coarse responses into full voxel-level predictions using a Mamba-VAE. Experiments on the Natural Scenes Dataset (NSD) demonstrate that our method achieves a Pearson correlation of 0.429 and an MSE of 0.261, outperforming all evaluated baselines including ridge regression and DINOv2 linear probes. Beyond predictive performance, causal branch-ablation experiments reveal an asymmetric specialization: the patch stream is specifically locked to early visual cortex (retinotopic regions), while the CLS stream contributes broader semantic context to higher-order areas -- a correspondence that holds causally, not merely correlationally. Cross-subject transfer experiments further show that the learned backbone generalizes across individuals with minimal per-subject adaptation, suggesting the model captures a shared, subject-agnostic visual representation.