mlsanityData to 14 Oct 2026

NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis

arXiv 2409.11836 ↗First posted 18 Sep 2024eess.IV cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Neural Transcoding Vision Transformer (modelname), a generative model designed to estimate high-resolution functional Magnetic Resonance Imaging (fMRI) samples from simultaneous Electroencephalography (EEG) data. A key feature of modelname is its Domain Matching (DM) sub-module which effectively aligns the latent EEG representations with those of fMRI volumes, enhancing the model's accuracy and reliability. Unlike previous methods that tend to struggle with fidelity and reproducibility of images, modelname addresses these challenges by ensuring methodological integrity and higher-quality reconstructions which we showcase through extensive evaluation on two benchmark datasets; modelname outperforms the current state-of-the-art by a significant margin in both cases, e.g. achieving a 10× reduction in RMSE and a 3.14× increase in SSIM on the Oddball dataset. An ablation study also provides insights into the contribution of each component to the model's overall effectiveness. This development is critical in offering a new approach to lessen the time and financial constraints typically linked with high-resolution brain imaging, thereby aiding in the swift and precise diagnosis of neurological disorders. Although it is not a replacement for actual fMRI but rather a step towards making such imaging more accessible, we believe that it represents a pivotal advancement in clinical practice and neuroscience research. Code is available at urlhttps://github.com/rom42pla/ntvit.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

31 of 56 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2016Layer Normalizationcited by 5,669
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
19 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2017Deep Image Priorcited by 1,098
  5. 2021Scaling Vision Transformerscited by 999
  6. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  7. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  8. 2017Self-Normalizing Neural Networkscited by 598
  9. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  10. 2021Early Convolutions Help Transformers See Bettercited by 396
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  12. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  13. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  14. 2016Medical Image Synthesis with Context-Aware Generative Adversarial Networkscited by 91
  15. 2017Deep MR to CT Synthesis using Unpaired Datacited by 86
  16. 2021A Generative Model for Raw Audio Using Transformer Architecturescited by 18
  17. 2018Why ReLU Units Sometimes Die: Analysis of Single-Unit Error Backpropagation in Neural Networkscited by 12
  18. 2020EEG to fMRI Synthesis: Is Deep Learning a candidate?cited by 8
  19. 2022EEG to fMRI Synthesis Benefits from Attentional Graphs of Electrode Relationshipscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025From Brainwaves to Brain Scans: A Robust Neural Network for EEG-to-fMRI Synthesiscited by 4
  2. 2025Spec2VolCAMU-Net: A Spectrogram-to-Volume Model for EEG-to-fMRI Reconstruction based on Multi-directional Time-Frequency Convolutional Attention Encoder and Vision-Mamba U-Netcited by 2
  3. 2026Physiology as Language: Translating Respiration to Sleep EEGcited by 0
  4. 2026MEL: Coordinate-Preserving EEG Tokenization for fMRI Translationcited by 0