mlsanityData to 14 Oct 2026

MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Models

arXiv 2607.22566 ↗First posted 31 May 2026cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

MedLoCoMo is a Medical Long-Context Memory benchmark for patient-specific clinical reasoning over multi-admission medical dialogue. Existing medical QA benchmarks largely test short context knowledge or single document grounding, leaving open whether LLMs can use, connect, and abstain over longitudinal patient histories. We build MedLoCoMo from deidentified MIMIC-IV and MIMIC-IV-Note records by constructing admission-level clinical packets, synthesizing grounded doctor-patient conversations, and generating evidence linked QA items over single-admission, cross-admission, and adversarial unanswerable settings. The benchmark contains 100 patient timelines averaging 1,669.8 turns, 29.7 sessions, and 74,512.2 tokens per conversation. Across the evaluated baselines, cross-admission reasoning is consistently harder than localized evidence use, even when models have long context windows or use external memory or retrieval methods. The code and MedLoCoMo benchmark release is available at https://github.com/leozzy13/MedLoCoMo for use and reproducibility.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

20 of 33 references.

References · 20

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  10. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
8 more
  1. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  3. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 135
  4. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  5. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  6. 2024MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokenscited by 13
  7. 2026MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?cited by 6
  8. 2026Synthesis and Evaluation of Long-term History-aware Medical Dialoguecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracycited by 0
  2. 2026Personalized State-Transition-Aware Memory for Clinical Agentscited by 0
  3. 2026Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoningcited by 0
  4. 2026MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Taskscited by 0