# MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data

arXiv: https://arxiv.org/abs/2606.20696 · HTML: https://mlsanity.com/paper/2606.20696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [MindBridge: A Cross-Subject Brain Decoding Framework](https://mlsanity.com/paper/2404.07850/) (2024)
- [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://mlsanity.com/paper/2410.07507/) (2024)
- [MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding](https://mlsanity.com/paper/2502.15786/) (2025)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [BrainLLM: Generative Language Decoding from Brain Recordings](https://mlsanity.com/paper/2311.09889/) (2023)
- [MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding](https://mlsanity.com/paper/2405.17720/) (2024)
- [LaVCa: LLM-assisted Visual Cortex Captioning](https://mlsanity.com/paper/2502.13606/) (2025)
- [BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models](https://mlsanity.com/paper/2406.07584/) (2024)
- [On Bessel's Correction: Unbiased Sample Variance, the Bariance, and a Novel Runtime-Optimized Estimator](https://mlsanity.com/paper/2503.22333/) (2025)
- [Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations](https://mlsanity.com/paper/2505.01670/) (2025)

## Abstract (arXiv)

Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.