mlsanityData to 7 Oct 2026

NeuGPT: Unified multi-modal Neural GPT

arXiv 2410.20916 ↗First posted 28 Oct 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper introduces NeuGPT, a groundbreaking multi-modal language generation model designed to harmonize the fragmented landscape of neural recording research. Traditionally, studies in the field have been compartmentalized by signal type, with EEG, MEG, ECoG, SEEG, fMRI, and fNIRS data being analyzed in isolation. Recognizing the untapped potential for cross-pollination and the adaptability of neural signals across varying experimental conditions, we set out to develop a unified model capable of interfacing with multiple modalities. Drawing inspiration from the success of pre-trained large models in NLP, computer vision, and speech processing, NeuGPT is architected to process a diverse array of neural recordings and interact with speech and text data. Our model mainly focus on brain-to-text decoding, improving SOTA from 6.94 to 12.92 on BLEU-1 and 6.93 to 13.06 on ROUGE-1F. It can also simulate brain signals, thereby serving as a novel neural interface. Code is available at hrefhttps://github.com/NeuSpeech/NeuGPTNeuSpeech/NeuGPT (https://github.com/NeuSpeech/NeuGPT) .

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 15 of its 37 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

15 of 37 references.

References · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  8. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  11. 2022Decoding speech perception from non-invasive brain recordingscited by 129
  12. 2021Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classificationcited by 67
3 more
  1. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 35
  2. 2024Are EEG-to-Text Models Working?cited by 32
  3. 2024MAD: Multi-Alignment MEG-to-Text Decodingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scalecited by 20
  2. 2025The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Datasetcited by 16
  3. 2025Unlocking Non-Invasive Brain-to-Textcited by 14
  4. 2025Foundation Models for Cross-Domain EEG Analysis Application: A Surveycited by 4
  5. 2025Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Datasetcited by 3
  6. 2026The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decodingcited by 1