# Estimating Redundancy in Clinical Text

arXiv: https://arxiv.org/abs/2105.11832 · HTML: https://mlsanity.com/paper/2105.11832/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 70 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit](https://mlsanity.com/paper/2010.01165/) (2020)
- [MedSTS: A Resource for Clinical Semantic Textual Similarity](https://mlsanity.com/paper/1808.09397/) (2018)
- [Deep Representation Learning of Electronic Health Records to Unlock Patient Stratification at Scale](https://mlsanity.com/paper/2003.06516/) (2020)
- [Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation](https://mlsanity.com/paper/1912.11975/) (2019)
- [MedCAT -- Medical Concept Annotation Tool](https://mlsanity.com/paper/1912.10166/) (2019)

## Built on by (most cited first, 10 of 10)

- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [Automated Clinical Coding: What, Why, and Where We Are?](https://mlsanity.com/paper/2203.11092/) (2022)
- [Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models](https://mlsanity.com/paper/2211.07126/) (2022)
- [Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs](https://mlsanity.com/paper/2212.08072/) (2022)
- [Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale](https://mlsanity.com/paper/2502.00943/) (2025)
- [Large Language Models for Medical Forecasting -- Foresight 2](https://mlsanity.com/paper/2412.10848/) (2024)
- [Clinical Text Deduplication Practices for Efficient Pretraining and Improved Clinical Tasks](https://mlsanity.com/paper/2312.09469/) (2023)
- [Content Reduction, Surprisal and Information Density Estimation for Long Documents](https://mlsanity.com/paper/2309.06009/) (2023)
- [Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care](https://mlsanity.com/paper/2403.19790/) (2024)
- [Application of Transformers based methods in Electronic Medical Records: A Systematic Literature Review](https://mlsanity.com/paper/2304.02768/) (2023)

## Abstract (arXiv)

The current mode of use of Electronic Health Record (EHR) elicits text redundancy. Clinicians often populate new documents by duplicating existing notes, then updating accordingly. Data duplication can lead to a propagation of errors, inconsistencies and misreporting of care. Therefore, quantifying information redundancy can play an essential role in evaluating innovations that operate on clinical narratives. This work is a quantitative examination of information redundancy in EHR notes. We present and evaluate two strategies to measure redundancy: an information-theoretic approach and a lexicosyntactic and semantic model. We evaluate the measures by training large Transformer-based language models using clinical text from a large openly available US-based ICU dataset and a large multi-site UK based Trust. By comparing the information-theoretic content of the trained models with open-domain language models, the language models trained using clinical text have shown ~1.5x to ~3x less efficient than open-domain corpora. Manual evaluation shows a high correlation with lexicosyntactic and semantic redundancy, with averages ~43 to ~65%.