# Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical texts

arXiv: https://arxiv.org/abs/2502.15996 · HTML: https://mlsanity.com/paper/2502.15996/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://mlsanity.com/paper/2104.06979/) (2021)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [A Survey on Contextual Embeddings](https://mlsanity.com/paper/2003.07278/) (2020)
- [LongEmbed: Extending Embedding Models for Long Context Retrieval](https://mlsanity.com/paper/2404.12096/) (2024)
- [An Extensive Data Processing Pipeline for MIMIC-IV](https://mlsanity.com/paper/2204.13841/) (2022)
- [MedSTS: A Resource for Clinical Semantic Textual Similarity](https://mlsanity.com/paper/1808.09397/) (2018)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [When Text Embedding Meets Large Language Model: A Comprehensive Survey](https://mlsanity.com/paper/2412.09165/) (2024)
- [Generalist embedding models are better at short-context clinical semantic search than specialized embedding models](https://mlsanity.com/paper/2401.01943/) (2024)
- [Improving Clinical Note Generation from Complex Doctor-Patient Conversation](https://mlsanity.com/paper/2408.14568/) (2024)
- [MvCo-DoT:Multi-View Contrastive Domain Transfer Network for Medical Report Generation](https://mlsanity.com/paper/2304.07465/) (2023)

## Built on by (most cited first, 1 of 1)

- [Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care](https://mlsanity.com/paper/2601.08503/) (2026)

## Abstract (arXiv)

We introduce a novel contextual embedding model med-gte-hybrid that was derived from the gte-large sentence transformer to extract information from unstructured clinical narratives. Our model tuning strategy for med-gte-hybrid combines contrastive learning and a denoising autoencoder. To evaluate the performance of med-gte-hybrid, we investigate several clinical prediction tasks in large patient cohorts extracted from the MIMIC-IV dataset, including Chronic Kidney Disease (CKD) patient prognosis, estimated glomerular filtration rate (eGFR) prediction, and patient mortality prediction. Furthermore, we demonstrate that the med-gte-hybrid model improves patient stratification, clustering, and text retrieval, thus outperforms current state-of-the-art models on the Massive Text Embedding Benchmark (MTEB). While some of our evaluations focus on CKD, our hybrid tuning of sentence transformers could be transferred to other medical domains and has the potential to improve clinical decision-making and personalised treatment pathways in various healthcare applications.