Spec2VolCAMU-Net: A Spectrogram-to-Volume Model for EEG-to-fMRI Reconstruction based on Multi-directional Time-Frequency Convolutional Attention Encoder and Vision-Mamba U-Net
On the map: State-space sequence models (S4, Mamba) → Physiological signal modeling
Abstract (arXiv)
High-resolution functional magnetic resonance imaging (fMRI) is essential for mapping human brain activity; however, it remains costly and logistically challenging. If comparable volumes could be generated directly from widely available scalp electroencephalography (EEG), advanced neuroimaging would become significantly more accessible. Existing EEG-to-fMRI generators rely on plain Convolutional Neural Networks (CNNs) that fail to capture cross-channel time-frequency cues or on heavy transformer/Generative Adversarial Network (GAN) decoders that strain memory and stability. To address these limitations, we propose Spec2VolCAMU-Net, a lightweight architecture featuring a Multi-directional Time-Frequency Convolutional Attention Encoder for rich feature extraction and a Vision-Mamba U-Net decoder that uses linear-time state-space blocks for efficient long-range spatial modelling. We frame the goal of this work as establishing a new state of the art in the spatial fidelity of single-volume reconstruction, a foundational prerequisite for the ultimate aim of generating temporally coherent fMRI time series. Trained end-to-end with a hybrid SSI-MSE loss, Spec2VolCAMU-Net achieves state-of-the-art fidelity on three public benchmarks, recording Structural Similarity Index (SSIM) of 0.693 on NODDI, 0.725 on Oddball and 0.788 on CN-EPFL, representing improvements of 14.5%, 14.9%, and 16.9% respectively over previous best SSIM scores. Furthermore, it achieves competitive Signal-to-Noise Ratio (PSNR) scores, particularly excelling on the CN-EPFL dataset with a 4.6% improvement over the previous best PSNR, thus striking a better balance in reconstruction quality. The proposed model is lightweight and efficient, making it suitable for real-time applications in clinical and research settings. The code is available at https://github.com/hdy6438/Spec2VolCAMU-Net.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Physiological signal modeling, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 14 of its 35 references that are in the map. First posted on arXiv 14 May 2025.
Builds on
14 of 35 references.
References · 14
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2016Improved Techniques for Training GANscited by 2,970
- 2022Diffusion Models in Vision: A Surveycited by 1,015
- 2024VMamba: Visual State Space Modelcited by 969
- 2018Which Training Methods for GANs do actually Converge?cited by 679
- 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 287
- 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
- 2024NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mappingcited by 11
- 2022EEG to fMRI Synthesis Benefits from Attentional Graphs of Electrode Relationshipscited by 7
- 2023Leveraging sinusoidal representation networks to predict fMRI signals from EEGcited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2023Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models
- 2022Inferring latent neural sources via deep transcoding of simultaneously acquired EEG and fMRI
- 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
- 2024NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties
- 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
- 2026Brain-to-Image Generation: Reconstructing Visual Stimuli from EEG using Generative Adversarial Networks
- 2024SWIM: Short-Window CNN Integrated with Mamba for EEG-Based Auditory Spatial Attention Decoding
- 2022SaleNet: A low-power end-to-end CNN accelerator for sustained attention level evaluation using EEG