# TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records

arXiv: https://arxiv.org/abs/2503.04176 · HTML: https://mlsanity.com/paper/2503.04176/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models](https://mlsanity.com/paper/2402.13064/) (2024)
- [AlpaCare:Instruction-tuned Large Language Models for Medical Application](https://mlsanity.com/paper/2310.14558/) (2023)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records](https://mlsanity.com/paper/2308.14089/) (2023)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2310.19975/) (2023)
- [Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Data](https://mlsanity.com/paper/2402.01713/) (2024)
- [Biomedical Visual Instruction Tuning with Clinician Preference Alignment](https://mlsanity.com/paper/2406.13173/) (2024)
- [Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing](https://mlsanity.com/paper/2401.00579/) (2023)
- [Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time](https://mlsanity.com/paper/2409.13338/) (2024)
- [Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data](https://mlsanity.com/paper/2410.18588/) (2024)

## Built on by (most cited first, 30 of 34)

- [Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning](https://mlsanity.com/paper/2505.24105/) (2025)
- [Reasoning LLMs in the Medical Domain: A Literature Survey](https://mlsanity.com/paper/2508.19097/) (2025)
- [Building the EHR Foundation Model via Next Event Prediction](https://mlsanity.com/paper/2509.25591/) (2025)
- [One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models](https://mlsanity.com/paper/2602.00541/) (2026)
- [Hypercube-Based Retrieval-Augmented Generation for Scientific Question-Answering](https://mlsanity.com/paper/2505.19288/) (2025)
- [AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization](https://mlsanity.com/paper/2601.13918/) (2026)
- [RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills](https://mlsanity.com/paper/2606.18203/) (2026)
- [CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series](https://mlsanity.com/paper/2607.09880/) (2026)
- [PMOA-TTS: Introducing the PubMed Open Access Textual Times Series Corpus](https://mlsanity.com/paper/2505.20323/) (2025)
- [Traj-CoA: Patient Trajectory Modeling via Chain-of-Agents for Lung Cancer Risk Prediction](https://mlsanity.com/paper/2510.10454/) (2025)
- [Large Language Models Lack Temporal Awareness of Medical Knowledge](https://mlsanity.com/paper/2605.13045/) (2026)
- [EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs](https://mlsanity.com/paper/2605.30637/) (2026)
- [PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models](https://mlsanity.com/paper/2606.24102/) (2026)
- [CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation](https://mlsanity.com/paper/2509.07325/) (2025)
- [Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus](https://mlsanity.com/paper/2604.24473/) (2026)
- [ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making](https://mlsanity.com/paper/2609.07601/) (2026)
- [Early Risk Prediction with Temporally and Contextually Grounded Clinical Language Processing](https://mlsanity.com/paper/2511.22038/) (2025)
- [Dense Feature Learning via Linear Structure Preservation in Medical Data](https://mlsanity.com/paper/2602.07706/) (2026)
- [ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents](https://mlsanity.com/paper/2604.02834/) (2026)
- [LiFT: How to Enable In-Context Learning for Longitudinal Modelling](https://mlsanity.com/paper/2604.16382/) (2026)
- [Traj-Evolve: A Self-Evolving Multi-Agent System for Patient Trajectory Modeling in Lung Cancer Early Detection](https://mlsanity.com/paper/2606.02812/) (2026)
- [VISTA Architect: A graph database-oriented health AI system demonstrated in multidisciplinary tumor boards](https://mlsanity.com/paper/2606.22692/) (2026)
- [Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.20560/) (2026)
- [PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise](https://mlsanity.com/paper/2608.02372/) (2026)
- [MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents](https://mlsanity.com/paper/2608.21810/) (2026)
- [MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain](https://mlsanity.com/paper/2609.04842/) (2026)
- [Anchoring Clinical Events in Time: UID-Preserving Multimodal Reconstruction and Source-Grounded Adjudication](https://mlsanity.com/paper/2609.13062/) (2026)
- [Grounded Adjudication of Variations across Extracted TimeLines (GAVEL): Comparing Clinical Timelines Against Their Case Reports](https://mlsanity.com/paper/2609.13475/) (2026)
- [CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives](https://mlsanity.com/paper/2609.19585/) (2026)
- [Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark](https://mlsanity.com/paper/2609.30027/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have emerged as promising tools for assisting in medical tasks, yet processing Electronic Health Records (EHRs) presents unique challenges due to their longitudinal nature. While LLMs' capabilities to perform medical tasks continue to improve, their ability to reason over temporal dependencies across multiple patient visits and time frames remains unexplored. We introduce TIMER (Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records), a framework that incorporate instruction-response pairs grounding to different parts of a patient's record as a critical dimension in both instruction evaluation and tuning for longitudinal clinical records. We develop TIMER-Bench, the first time-aware benchmark that evaluates temporal reasoning capabilities over longitudinal EHRs, as well as TIMER-Instruct, an instruction-tuning methodology for LLMs to learn reasoning over time. We demonstrate that models fine-tuned with TIMER-Instruct improve performance by 7.3% on human-generated benchmarks and 9.2% on TIMER-Bench, indicating that temporal instruction-tuning improves model performance for reasoning over EHR.