# Time Matters: Examine Temporal Effects on Biomedical Language Models

arXiv: https://arxiv.org/abs/2407.17638 · HTML: https://mlsanity.com/paper/2407.17638/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature](https://mlsanity.com/paper/2405.04819/) (2024)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Temporal Effects on Pre-trained Models for Language Processing Tasks](https://mlsanity.com/paper/2111.12790/) (2021)
- [BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition](https://mlsanity.com/paper/2009.09223/) (2020)
- [Token Imbalance Adaptation for Radiology Report Generation](https://mlsanity.com/paper/2304.09185/) (2023)
- [Thought Graph: Generating Thought Process for Biological Reasoning](https://mlsanity.com/paper/2403.07144/) (2024)
- [Length-Aware Multi-Kernel Transformer for Long Document Classification](https://mlsanity.com/paper/2405.07052/) (2024)
- [Evaluating Model Performance in Medical Datasets Over Time](https://mlsanity.com/paper/2305.13426/) (2023)

## Built on by (most cited first, 7 of 7)

- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts](https://mlsanity.com/paper/2502.08825/) (2025)
- [Predicting 30-Day Hospital Readmission in Medicare Patients: Insights from an LSTM Deep Learning Model](https://mlsanity.com/paper/2410.17545/) (2024)
- [Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memory](https://mlsanity.com/paper/2605.03228/) (2026)
- [Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models](https://mlsanity.com/paper/2412.17803/) (2024)
- [Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity](https://mlsanity.com/paper/2507.05816/) (2025)
- [Knowledge-driven Augmentation and Retrieval for Integrative Temporal Adaptation](https://mlsanity.com/paper/2604.22098/) (2026)

## Abstract (arXiv)

Time roots in applying language models for biomedical applications: models are trained on historical data and will be deployed for new or future data, which may vary from training data. While increasing biomedical tasks have employed state-of-the-art language models, there are very few studies have examined temporal effects on biomedical models when data usually shifts across development and deployment. This study fills the gap by statistically probing relations between language model performance and data shifts across three biomedical tasks. We deploy diverse metrics to evaluate model performance, distance methods to measure data drifts, and statistical methods to quantify temporal effects on biomedical language models. Our study shows that time matters for deploying biomedical language models, while the degree of performance degradation varies by biomedical tasks and statistical quantification approaches. We believe this study can establish a solid benchmark to evaluate and assess temporal effects on deploying biomedical language models.