# Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

arXiv: https://arxiv.org/abs/2411.17073 · HTML: https://mlsanity.com/paper/2411.17073/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images](https://mlsanity.com/paper/1806.01963/) (2018)
- [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images](https://mlsanity.com/paper/1907.09478/) (2019)
- [HistoCartography: A Toolkit for Graph Analytics in Digital Pathology](https://mlsanity.com/paper/2107.10073/) (2021)

## Built on by (most cited first, 1 of 1)

- [Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation](https://mlsanity.com/paper/2506.20869/) (2025)

## Abstract (arXiv)

Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).