# ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling

arXiv: https://arxiv.org/abs/2609.37587 · HTML: https://mlsanity.com/paper/2609.37587/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction](https://mlsanity.com/paper/2505.23416/) (2025)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combination](https://mlsanity.com/paper/1809.01852/) (2018)
- [EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records](https://mlsanity.com/paper/2401.07128/) (2024)
- [Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models](https://mlsanity.com/paper/2308.15022/) (2023)
- [SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinations](https://mlsanity.com/paper/2105.02711/) (2021)
- [Compressed Context Memory For Online Language Model Interaction](https://mlsanity.com/paper/2312.03414/) (2023)
- [Fast KV Compaction via Attention Matching](https://mlsanity.com/paper/2602.16284/) (2026)
- [EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis](https://mlsanity.com/paper/2510.25628/) (2025)
- [A multimodal and temporal foundation model for virtual patient representations at healthcare system scale](https://mlsanity.com/paper/2604.18570/) (2026)
- [Large Language Models for Medical Forecasting -- Foresight 2](https://mlsanity.com/paper/2412.10848/) (2024)

## Abstract (arXiv)

Longitudinal electronic health record (EHR) modeling requires integrating new visits with an expanding patient history. Yet the continual accumulation of clinical information imposes increasing computational and memory costs on large language models (LLMs) when they process and retain complete patient histories. A practical alternative is visit-wise recurrent compression, which incorporates each incoming visit into a compact, continually updated patient memory. However, under a fixed memory budget, successive updates must integrate new information without progressively losing critical historical evidence needed to subsequent tasks. To address this challenge, we introduce Recurrent Longitudinal Memory (ReLMem), a framework that learns to maintain fixed-capacity patient memory for efficient downstream prediction with a frozen LLM. ReLMem equips this LLM with lightweight compression adapters to recurrently update the memory from its previous state and each incoming visit, without rereading earlier records. Specifically, we develop a multi-granularity optimization strategy to preserve task-relevant information throughout recurrent updates and support downstream prediction from the final memory. The intermediate supervision aligns attention outputs from compressed memory and the full history under identical queries, while prediction supervision minimizes cross-entropy with ground truth answers conditioned on the final memory. On EHR-based medication prediction, ReLMem approaches the F1 scores of full-history baseline while reducing average retained historical storage by 97.1%. Under the same memory budget, it improves macro- and micro-F1 over the strongest compressed-memory baseline by 4.66 and 4.75 percentage points, respectively. These results highlight the value of learning recurrent patient memory for efficient longitudinal EHR modeling.