# UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

arXiv: https://arxiv.org/abs/2307.05355 · HTML: https://mlsanity.com/paper/2307.05355/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [Brain2Word: Decoding Brain Activity for Language Generation](https://mlsanity.com/paper/2009.04765/) (2020)
- [Progress in Brain Computer Interfaces: Challenges and Trends](https://mlsanity.com/paper/1901.03442/) (2019)

## Built on by (most cited first, 22 of 22)

- [Are EEG-to-Text Models Working?](https://mlsanity.com/paper/2405.06459/) (2024)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder](https://mlsanity.com/paper/2402.17433/) (2024)
- [From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery](https://mlsanity.com/paper/2309.05203/) (2023)
- [NeuSpeech: Decode Neural signal as Speech](https://mlsanity.com/paper/2403.01748/) (2024)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [BrainLLM: Generative Language Decoding from Brain Recordings](https://mlsanity.com/paper/2311.09889/) (2023)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation](https://mlsanity.com/paper/2410.14971/) (2024)
- [Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding](https://mlsanity.com/paper/2312.10987/) (2023)
- [BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2502.15172/) (2025)
- [What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval](https://mlsanity.com/paper/2605.24524/) (2026)
- [MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities](https://mlsanity.com/paper/2403.17516/) (2024)
- [Query Augmentation by Decoding Semantics from Brain Signals](https://mlsanity.com/paper/2402.15708/) (2024)
- [Language Reconstruction with Brain Predictive Coding from fMRI Data](https://mlsanity.com/paper/2405.11597/) (2024)
- [Decoding Linguistic Representations of Human Brain](https://mlsanity.com/paper/2407.20622/) (2024)
- [Bridging Brain with Foundation Models through Self-Supervised Learning](https://mlsanity.com/paper/2506.16009/) (2025)
- [Decoding Continuous Character-based Language from Non-invasive Brain Recordings](https://mlsanity.com/paper/2403.11183/) (2024)
- [Volume-Wise Task fMRI Decoding with Deep Learning:Enhancing Temporal Resolution and Cognitive Function Analysis](https://mlsanity.com/paper/2503.01925/) (2025)
- [Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging](https://mlsanity.com/paper/2505.22150/) (2025)
- [EmoMind: Decoding Affective Captions from Human Brain fMRI](https://mlsanity.com/paper/2605.16739/) (2026)
- [Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark](https://mlsanity.com/paper/2607.18749/) (2026)

## Abstract (arXiv)

Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEGto-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.