mlsanityData to 7 Oct 2026

ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes

arXiv 2403.05795 ↗First posted 9 Mar 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

The advancement of natural language processing (NLP) systems in healthcare hinges on language model ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical language models and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

30 of 47 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2019Publicly Available Clinical BERT Embeddingscited by 995
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  11. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  12. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
18 more
  1. 2023Do We Still Need Clinical Language Models?cited by 97
  2. 2019ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Networkcited by 73
  3. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  4. 2019Document-Level N-ary Relation Extraction with Multiscale Representation Learningcited by 52
  5. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  6. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  7. 2023Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understandingcited by 40
  8. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
  9. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  10. 2023Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Healthcited by 25
  11. 2023Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncologycited by 24
  12. 2022PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learningcited by 22
  13. 2022MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Scorecited by 20
  14. 2022Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRscited by 11
  15. 2019Clinical Concept Extraction for Document-Level Codingcited by 7
  16. 2022Hierarchical BERT for Medical Document Understandingcited by 4
  17. 2023LongBoX: Evaluating Transformers on Long-Sequence Clinical Taskscited by 3
  18. 2024LocalTweets to LocalHealth: A Mental Health Surveillance Framework Based on Twitter Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 7; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  2. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  3. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  4. 2024A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyondcited by 16
  5. 2025Foundation models for electronic health records: representation dynamics and transferabilitycited by 11
  6. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  7. 2024SELD-Mamba: Selective State-Space Model for Sound Event Localization and Detection with Source Distance Estimationcited by 6
  8. 2025MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncologycited by 6
  9. 2025Reward Hacking Mitigation using Verifiable Composite Rewardscited by 5
  10. 2024FACTS: A Factored State-Space Framework For World Modellingcited by 4
  11. 2024Synth-SBDH: A Synthetic Dataset of Social and Behavioral Determinants of Health for Clinical Textcited by 3
  12. 2025Quantifying surprise in clinical care: Detecting highly informative events in electronic health records with foundation modelscited by 3
10 more
  1. 2025Clinical trial cohort selection using Large Language Models on n2c2 Challengescited by 2
  2. 2024SPMamba: State-space model is all you need in speech separationcited by 1
  3. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  4. 2024MASV: Speaker Verification with Global and Local Context Mambacited by 1
  5. 2025Comparing representations of long clinical texts for the task of patient note-identificationcited by 1
  6. 2026Omni-directional attention mechanism based on Mamba for speech separationcited by 1
  7. 2024Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERTcited by 0
  8. 2026Improving Patient Subtyping on Longitudinal Data using Representations from Mamba-based Architecturecited by 0
  9. 2026AraSSM: A bidirectional state-space encoder for Arabic masked language modelingcited by 0
  10. 2026OverdoseMoE: A Multi-Expert Framework for Opioid Overdose Risk Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rephrasing Electronic Health Records for Pretraining Clinical Language Models
  2. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing
  3. 2025CLI-RAG: A Retrieval-Augmented Framework for Clinically Structured and Context Aware Text Generation with LLMs
  4. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
  5. 2025Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction
  6. 2025Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings
  7. 2024Large Language Models Struggle in Token-Level Clinical Named Entity Recognition
  8. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text