# MedSTS: A Resource for Clinical Semantic Textual Similarity

arXiv: https://arxiv.org/abs/1808.09397 · HTML: https://mlsanity.com/paper/1808.09397/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 30 of 39)

- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation](https://mlsanity.com/paper/2102.01818/) (2021)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)
- [BioLORD: Learning Ontological Representations from Definitions (for Biomedical Concepts and their Textual Descriptions)](https://mlsanity.com/paper/2210.11892/) (2022)
- [Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions](https://mlsanity.com/paper/2404.09135/) (2024)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Estimating Redundancy in Clinical Text](https://mlsanity.com/paper/2105.11832/) (2021)
- [Collective Human Opinions in Semantic Textual Similarity](https://mlsanity.com/paper/2308.04114/) (2023)
- [Improving the robustness and accuracy of biomedical language models through adversarial training](https://mlsanity.com/paper/2111.08529/) (2021)
- [BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insights](https://mlsanity.com/paper/2311.16075/) (2023)
- [Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words](https://mlsanity.com/paper/2209.06453/) (2022)
- [Compositional Zero-Shot Domain Transfer with Text-to-Text Models](https://mlsanity.com/paper/2303.13386/) (2023)
- [BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning](https://mlsanity.com/paper/2410.18955/) (2024)
- [Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications](https://mlsanity.com/paper/2209.09062/) (2022)
- [Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contexts](https://mlsanity.com/paper/2308.00157/) (2023)
- [Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selection](https://mlsanity.com/paper/2308.09892/) (2023)
- [Rethinking STS and NLI in Large Language Models](https://mlsanity.com/paper/2309.08969/) (2023)
- [SPICED: News Similarity Detection Dataset with Multiple Topics and Complexity Levels](https://mlsanity.com/paper/2309.13080/) (2023)
- [CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports](https://mlsanity.com/paper/2505.17265/) (2025)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Diagnosable ColBERT: Debugging Late-Interaction Retrieval Models Using a Learned Latent Space as Reference](https://mlsanity.com/paper/2604.19566/) (2026)

## Abstract (arXiv)

The wide adoption of electronic health records (EHRs) has enabled a wide range of applications leveraging EHR data. However, the meaningful use of EHR data largely depends on our ability to efficiently extract and consolidate information embedded in clinical text where natural language processing (NLP) techniques are essential. Semantic textual similarity (STS) that measures the semantic similarity between text snippets plays a significant role in many NLP applications. In the general NLP domain, STS shared tasks have made available a huge collection of text snippet pairs with manual annotations in various domains. In the clinical domain, STS can enable us to detect and eliminate redundant information that may lead to a reduction in cognitive burden and an improvement in the clinical decision-making process. This paper elaborates our efforts to assemble a resource for STS in the medical domain, MedSTS. It consists of a total of 174,629 sentence pairs gathered from a clinical corpus at Mayo Clinic. A subset of MedSTS (MedSTS_ann) containing 1,068 sentence pairs was annotated by two medical experts with semantic similarity scores of 0-5 (low to high similarity). We further analyzed the medical concepts in the MedSTS corpus, and tested four STS systems on the MedSTS_ann corpus. In the future, we will organize a shared task by releasing the MedSTS_ann corpus to motivate the community to tackle the real world clinical problems.