mlsanityData to 7 Oct 2026

Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering

arXiv 2510.18691 ↗First posted 21 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA. Our comprehensive approach considers a range of settings based on content inclusion of varying size and relevance, LLM models of different capabilities and a variety of datasets across task formulations. We reveal insights on model size effects and their limitations, underlying memorization issues and the benefits of reasoning models, while demonstrating the value and challenges of leveraging the full long patient's context. Importantly, we examine the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension, showcasing best settings in single versus multi-document QA datasets. We shed light into some of the evaluation aspects using a multi-faceted approach uncovering common metric challenges. Our quantitative analysis reveals challenging cases where RAG excels while still showing limitations in cases requiring temporal reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 97 of its 127 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

97 of 127 references.

References · 97

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
38 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2019Publicly Available Clinical BERT Embeddingscited by 995
  13. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  14. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  15. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  16. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  17. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  18. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  19. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  20. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  21. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  22. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  23. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  24. 2023Adapting Language Models to Compress Contextscited by 350
  25. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  26. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  27. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  28. 2023Improving Text Embeddings with Large Language Modelscited by 295
  29. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  30. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  31. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  32. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  33. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  34. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  35. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  36. 2022Can large language models reason about medical questions?cited by 194
  37. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  38. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182

50 most cited of 97 All 97 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering
  2. 2024Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering
  3. 2025Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
  4. 2026Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
  5. 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts
  6. 2026A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
  7. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability
  8. 2025ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems