# CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding

arXiv: https://arxiv.org/abs/2402.08994 · HTML: https://mlsanity.com/paper/2402.08994/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Multi-label Co-regularization for Semi-supervised Facial Action Unit Recognition](https://mlsanity.com/paper/1910.11012/) (2019)
- [Deep Recurrent Encoder: A scalable end-to-end network to model brain signals](https://mlsanity.com/paper/2103.02339/) (2021)
- [A Convolutional Autoencoder for Multi-Subject fMRI Data Aggregation](https://mlsanity.com/paper/1608.04846/) (2016)
- [A shared neural encoding model for the prediction of subject-specific fMRI response](https://mlsanity.com/paper/2006.15802/) (2020)
- [Deep Hyperalignment](https://mlsanity.com/paper/1710.03923/) (2017)
- [Graph-Based Decoding Model for Functional Alignment of Unaligned fMRI Data](https://mlsanity.com/paper/1905.05468/) (2019)

## Built on by (most cited first, 13 of 13)

- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [NeuroBind: Towards Unified Multimodal Representations for Neural Signals](https://mlsanity.com/paper/2407.14020/) (2024)
- [Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity](https://mlsanity.com/paper/2405.03280/) (2024)
- [Toward Generalizing Visual Brain Decoding to Unseen Subjects](https://mlsanity.com/paper/2410.14445/) (2024)
- [MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding](https://mlsanity.com/paper/2505.15946/) (2025)
- [A Survey of fMRI to Image Reconstruction](https://mlsanity.com/paper/2502.16861/) (2025)
- [ViEEG: Hierarchical Visual Neural Representation for EEG Brain Decoding](https://mlsanity.com/paper/2505.12408/) (2025)
- [Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing](https://mlsanity.com/paper/2503.10663/) (2025)
- [Does CLIP perceive art the same way we do?](https://mlsanity.com/paper/2505.05229/) (2025)
- [Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry](https://mlsanity.com/paper/2605.20496/) (2026)
- [FedMinds: Privacy-Preserving Personalized Brain Visual Decoding](https://mlsanity.com/paper/2409.02044/) (2024)
- [HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchies](https://mlsanity.com/paper/2605.09392/) (2026)
- [BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language](https://mlsanity.com/paper/2606.30319/) (2026)

## Abstract (arXiv)

The study of decoding visual neural information faces challenges in generalizing single-subject decoding models to multiple subjects, due to individual differences. Moreover, the limited availability of data from a single subject has a constraining impact on model performance. Although prior multi-subject decoding methods have made significant progress, they still suffer from several limitations, including difficulty in extracting global neural response features, linear scaling of model parameters with the number of subjects, and inadequate characterization of the relationship between neural responses of different subjects to various stimuli. To overcome these limitations, we propose a CLIP-guided Multi-sUbject visual neural information SEmantic Decoding (CLIP-MUSED) method. Our method consists of a Transformer-based feature extractor to effectively model global neural representations. It also incorporates learnable subject-specific tokens that facilitates the aggregation of multi-subject data without a linear increase of parameters. Additionally, we employ representational similarity analysis (RSA) to guide token representation learning based on the topological relationship of visual stimuli in the representation space of CLIP, enabling full characterization of the relationship between neural responses of different subjects under different stimuli. Finally, token representations are used for multi-subject semantic decoding. Our proposed method outperforms single-subject decoding methods and achieves state-of-the-art performance among the existing multi-subject methods on two fMRI datasets. Visualization results provide insights into the effectiveness of our proposed method. Code is available at https://github.com/CLIP-MUSED/CLIP-MUSED.