mlsanityData to 7 Oct 2026

EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping

arXiv 2409.07480 ↗First posted 2 Sep 2024eess.SP cs.AI cs.LGcited by 6

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Multimodal language modeling has enabled breakthroughs for representation learning, yet remains unexplored in the realm of functional brain data for clinical phenotyping. This paper pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to combine multimodal alignment in this novel domain with timeseries cropping and text segmentation, enabling an extension based on multiple instance learning to alleviate misalignment between irrelevant EEG or text segments. Our multimodal models significantly improve over EEG-only models across four clinical evaluations and for the first time enable zero-shot classification as well as retrieval of both neural signals and reports. In sum, these results highlight the potential of ELMs, representing significant progress for clinical applications.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 65 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

32 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training11 Oct
  2. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization17 Jul
  3. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation7 Jun

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  11. 2019Publicly Available Clinical BERT Embeddingscited by 995
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
17 more
  1. 2017Large Batch Training of Convolutional Networkscited by 529
  2. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  4. 2020Understanding the Behaviour of Contrastive Losscited by 397
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  6. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
  7. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  8. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  9. 2020Uncovering the structure of clinical EEG signals with self-supervised learningcited by 109
  10. 2018The Temple University Hospital Seizure Detection Corpuscited by 80
  11. 2020Subject-Aware Contrastive Learning for Biosignalscited by 75
  12. 2023Exploring Vision-Language Models for Imbalanced Learningcited by 35
  13. 2020Machine-Learning-Based Diagnostics of EEG Pathologycited by 32
  14. 2022Data augmentation for learning predictive models on EEG: a systematic comparisoncited by 32
  15. 2018ChronoNet: A Deep Recurrent Neural Network for Abnormal EEG Identificationcited by 27
  16. 2023ConvNets Match Vision Transformers at Scalecited by 16
  17. 2023Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Large Language Models for EEG: A Comprehensive Survey and Taxonomycited by 4
  2. 2026Neural Signals Generate Clinical Notes in the Wildcited by 4
  3. 2026KAST-BAR: Knowledge-Anchored Semantically-Dynamic Topology Brain Autoregressive Modeling for Universal Neural Interpretationcited by 2
  4. 2025Bridging Brain with Foundation Models through Self-Supervised Learningcited by 1
  5. 2026NeuroNarrator: A Generalist EEG-to-Text Foundation Model for Clinical Interpretation via Spectro-Spatial Grounding and Temporal State-Space Reasoningcited by 0
  6. 2026Flow Matching with In-Context Priors for Out-of-Distribution Brain Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EEG2TEXT-CN: An Exploratory Study of Open-Vocabulary Chinese Text-EEG Alignment via Large Language Model and Contrastive Learning on ChineseEEG
  2. 2026Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding
  3. 2024Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)
  4. 2024SEE: Semantically Aligned EEG-to-Text Translation
  5. 2025DELTA: Language Diffusion-based EEG-to-Text Architecture
  6. 2024EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation
  7. 2026RAG-based EEG-to-Text Translation Using Deep Learning and LLMs
  8. 2026BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding