One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligence
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Medical AI, including clinical language models, vision-language models, and multimodal health record models, already summarizes notes, answers questions, and supports decisions. Their adaptation to new populations, specialties, or care settings often relies on fine-tuning, prompting, or retrieval from external knowledge bases. These strategies can scale poorly and risk contextual errors: outputs that appear plausible but miss critical patient or situational information. We envision context switching as a solution. Context switching adjusts model reasoning at inference without retraining. Generative models can tailor outputs to patient biology, care setting, or disease. Multimodal models can reason on notes, laboratory results, imaging, and genomics, even when some data are missing or delayed. Agent models can coordinate tools and roles based on tasks and users. In each case, context switching enables medical AI to adapt across specialties, populations, and geographies. It requires advances in data design, model architectures, and evaluation frameworks, and establishes a foundation for medical AI that scales to infinitely many contexts while remaining reliable and suited to real-world care.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 111 references that are in the map. First posted on arXiv 11 Jun 2025.
Builds on
49 of 111 references.
References · 49
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
- 2022Large Language Models Encode Clinical Knowledgecited by 1,830
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
- 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
- 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
- 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
- 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
37 more
- 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
- 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
- 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
- 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
- 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
- 2025BIG-Bench Extra Hardcited by 125
- 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
- 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
- 20234M: Massively Multimodal Masked Modelingcited by 120
- 2024Empowering Biomedical Discovery with AI Agentscited by 118
- 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
- 2025SycEval: Evaluating LLM Sycophancycited by 108
- 2024Zero Shot Health Trajectory Prediction Using Transformercited by 77
- 2023Unleashing the potential of prompt engineering for large language modelscited by 72
- 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 70
- 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
- 2025TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Toolscited by 59
- 2022Multitask Vision-Language Prompt Tuningcited by 51
- 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
- 2025Reasoning Language Models: A Blueprintcited by 42
- 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
- 2023Conformal Nucleus Samplingcited by 33
- 2023A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Recordscited by 31
- 2019Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoderscited by 27
- 2024Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Modelscited by 25
- 2025Knowing You Don't Know: Learning When to Continue Search in Multi-round RAG through Self-Practicingcited by 25
- 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language modelscited by 20
- 2024KGARevion: An AI Agent for Knowledge-Intensive Biomedical QAcited by 20
- 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 15
- 2022RepsNet: Combining Vision with Language for Automated Medical Reportscited by 14
- 2024PDL: A Declarative Prompt Programming Languagecited by 14
- 2024Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learningcited by 13
- 2021Structure Inducing Pre-Trainingcited by 10
- 2024Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusioncited by 9
- 2025AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradientscited by 7
- 2025DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradientscited by 5
- 2025Multimodal AI predicts clinical outcomes of drug combinations from preclinical datacited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Medical Chatbots: Meddollina and the Rise of Continuous Clinical Intelligence
- 2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios
- 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
- 2026Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translation
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy
- 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
- 2026Generated Context versus Governed State: Functional Conditions for Accountable Longitudinal Clinical Reasoning
- 2026MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents