# Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI

arXiv: https://arxiv.org/abs/2605.01240 · HTML: https://mlsanity.com/paper/2605.01240/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Physiological signal modeling, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 24 of its 84 references that are in the map.
- First posted on arXiv 2 May 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Physiological signal modeling](https://mlsanity.com/ssm/physiological-signal-modeling/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [An Empirical Study of Mamba-based Language Models](https://mlsanity.com/paper/2406.07887/) (2024)
- [Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks](https://mlsanity.com/paper/2402.04248/) (2024)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking](https://mlsanity.com/paper/2409.19407/) (2024)
- [Learning by Reconstruction Produces Uninformative Features For Perception](https://mlsanity.com/paper/2402.11337/) (2024)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [Resting state fMRI functional connectivity-based classification using a convolutional neural network architecture](https://mlsanity.com/paper/1707.06682/) (2017)
- [Whole MILC: generalizing learned dynamics across tasks, datasets, and populations](https://mlsanity.com/paper/2007.16041/) (2020)
- [TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2503.24067/) (2025)
- [BrainMAE: A Region-aware Self-supervised Learning Framework for Brain Signals](https://mlsanity.com/paper/2406.17086/) (2024)
- [Early Disease Stage Characterization in Parkinson's Disease from Resting-state fMRI Data Using a Long Short-term Memory Network](https://mlsanity.com/paper/2202.12715/) (2022)
- [BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data](https://mlsanity.com/paper/2506.22591/) (2025)
- [Region-Aware Reconstruction Strategy for Pre-training fMRI Foundation Model](https://mlsanity.com/paper/2511.00443/) (2025)

## Abstract (arXiv)

Self-supervised pretraining is promising for large-scale neuroimaging, yet the impact of region-aware masking and hybrid sequence modeling remains underexplored. In this work, we introduce Rhamba, a region-aware pretraining framework that integrates anatomically guided masking with hybrid Attention-Mamba architectures for resting state functional magnetic resonance imaging (fMRI) analysis. Models were pretrained on the ABIDE dataset using region-aligned patch embeddings and three masking strategies (Any, Majority, and Pure) with increasing spatial specificity. We evaluated four architectural variants: a Mamba only model, an Alternate architecture with interleaved Mamba and Attention blocks, and two hybrid encoder-decoder configurations (Attention-Mamba (AM) and Mamba-Attention (MA)). The pretrained models were fine-tuned on downstream classification tasks using the COBRE and ADHD-200 datasets for schizophrenia and attention-deficit/hyperactivity disorder discrimination. We employed Integrated Gradients, an explainable AI method, to identify the brain regions contributing to model predictions. Masking strategy strongly influenced reconstruction behavior, with reconstruction loss following a consistent ordering (Any > Majority > Pure). However, this trend did not directly translate into downstream performance, where differences were modest and dataset-dependent. The hybrid architecture with the MA configuration achieved the highest average AUROC across both datasets, and Rhamba outperformed state-of-the-art methods in comparative evaluation. Region-wise analysis showed that peak performance depends on the interaction between masking strategy and architecture rather than a single dominant configuration. Overall, Rhamba offers a flexible framework for balancing interpretability, scalability, and performance in large-scale fMRI representation learning.