# FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

arXiv: https://arxiv.org/abs/2609.31204 · HTML: https://mlsanity.com/paper/2609.31204/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking](https://mlsanity.com/paper/2409.19407/) (2024)
- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [Spherical U-Net on Cortical Surfaces: Methods and Applications](https://mlsanity.com/paper/1904.00906/) (2019)
- [BrainMass: Advancing Brain Network Analysis for Diagnosis with Large-scale Self-Supervised Learning](https://mlsanity.com/paper/2403.01433/) (2024)
- [Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis](https://mlsanity.com/paper/2203.16414/) (2022)
- [Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens](https://mlsanity.com/paper/2509.24693/) (2025)
- [Omni-fMRI: A Universal Atlas-Free fMRI Foundation Model](https://mlsanity.com/paper/2601.23090/) (2026)
- [Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining](https://mlsanity.com/paper/2604.12683/) (2026)
- [SLIM-Brain: A Data- and Training-Efficient Foundation Model for fMRI Data Analysis](https://mlsanity.com/paper/2512.21881/) (2025)
- [MinD-3D++: Advancing fMRI-Based 3D Reconstruction with High-Quality Textured Mesh Generation and a Comprehensive Dataset](https://mlsanity.com/paper/2409.11315/) (2024)
- [FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI](https://mlsanity.com/paper/2606.11500/) (2026)
- [Voxel-Level Brain States Prediction Using Swin Transformer](https://mlsanity.com/paper/2506.11455/) (2025)
- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [DCA: Graph-Guided Deep Embedding Clustering for Brain Atlases](https://mlsanity.com/paper/2509.01426/) (2025)
- [Large Connectome Model: An fMRI Foundation Model of Brain Connectomes Empowered by Brain-Environment Interaction in Multitask Learning Landscape](https://mlsanity.com/paper/2510.18910/) (2025)
- [Spectral Characterization of functional MRI data on voxel-resolution cortical graphs](https://mlsanity.com/paper/1910.09507/) (2019)
- [Improved Multiscale Structural Mapping with Supervertex Vision Transformer for the Detection of Alzheimer's Disease Neurodegeneration](https://mlsanity.com/paper/2604.14837/) (2026)
- [X-SiT: Inherently Interpretable Surface Vision Transformers for Dementia Diagnosis](https://mlsanity.com/paper/2506.20267/) (2025)

## Abstract (arXiv)

Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.