mlsanityData to 14 Oct 2026

Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning

arXiv 2508.02258 ↗First posted 4 Aug 2025cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 2 more
Abstract (arXiv)

Although Vision Language Models (VLMs) have shown strong generalization in medical imaging, pathology presents unique challenges due to ultra-high resolution, complex tissue structures, and nuanced clinical semantics. These factors make pathology VLMs prone to hallucinations, i.e., generating outputs inconsistent with visual evidence, which undermines clinical trust. Existing RAG approaches in this domain largely depend on text-based knowledge bases, limiting their ability to leverage diagnostic visual cues. To address this, we propose Patho-AgenticRAG, a multimodal RAG framework with a database built on page-level embeddings from authoritative pathology textbooks. Unlike traditional text-only retrieval systems, it supports joint text-image search, enabling direct retrieval of textbook pages that contain both the queried text and relevant visual cues, thus avoiding the loss of critical image-based information. Patho-AgenticRAG also supports reasoning, task decomposition, and multi-turn search interactions, improving accuracy in complex diagnostic scenarios. Experiments show that Patho-AgenticRAG significantly outperforms existing multimodal models in complex pathology tasks like multiple-choice diagnosis and visual question answering. Our project is available at the Patho-AgenticRAG repository: https://github.com/Wenchuan-Zhang/Patho-AgenticRAG.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 43 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

38 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025RARL: Improving Medical VLM Reasoning and Generalization with Reinforcement Learning and LoRA under Data and Hardware Constraints7 Jun
  2. 2025Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models20 May
  3. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner16 May
  4. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  5. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  6. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology29 Jan

Other references · 32

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  6. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  9. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  10. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  11. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  12. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
20 more
  1. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  2. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  3. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  5. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 90
  6. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  7. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  8. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  9. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedbackcited by 32
  10. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 29
  11. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 27
  12. 2024REAPER: Reasoning based Retrieval Planning for Complex RAG Systemscited by 16
  13. 2024Agentic Retrieval-Augmented Generation for Time Series Analysiscited by 14
  14. 2024Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathologycited by 14
  15. 2024Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similaritycited by 12
  16. 2025Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoningcited by 4
  17. 2024Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reportscited by 2
  18. 2025Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQAcited by 2
  19. 2025Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLPcited by 1
  20. 2025AI-VaxGuide: An Agentic RAG-Based LLM for Vaccination Decisionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  2. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  3. 2025CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretationcited by 7
  4. 2026PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQAcited by 7
  5. 2025SurvAgent: Hierarchical CoT-Enhanced Case Banking and Dichotomy-Based Multi-Agent System for Multimodal Survival Predictioncited by 5
  6. 2025AT-CXR: Uncertainty-Aware Agentic Triage for Chest X-rayscited by 4
  7. 2026DramaDirector: Geometry-Guided Short Drama Generationcited by 2
  8. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  9. 2026MedVerse: Efficient and Reliable Medical Reasoning via DAG-Structured Parallel Executioncited by 1
  10. 2026PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMscited by 1
  11. 2026Toward Auditable Neuro-Symbolic Reasoning in Pathology: SQL as an Explicit Trace of Evidencecited by 0
  12. 2026SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Taskscited by 0
1 more
  1. 2026Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PathFound: An Agentic Multimodal Model Activating Evidence-seeking Pathological Diagnosis
  2. 2026ReinPath: A Multimodal Reinforcement Learning Approach for Pathology
  3. 2025PathVLM-R1: A Reinforcement Learning-Driven Reasoning Model for Pathology Visual-Language Tasks
  4. 2026PathoSage: Towards Multi-Source Evidence Adjudication in Pathology via Experience-Aware Agentic Workflow
  5. 2025Proactive Reasoning-with-Retrieval Framework for Medical Multimodal Large Language Models
  6. 2026A Multimodal Agentic Pathology Co-pilot via Evidence Grounded Reasoning
  7. 2025PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning
  8. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology