Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Pathological diagnosis is vital for determining disease characteristics, guiding treatment, and assessing prognosis, relying heavily on detailed, multi-scale analysis of high-resolution whole slide images (WSI). However, existing large vision-language models (LVLMs) are limited by input resolution constraints, hindering their efficiency and accuracy in pathology image analysis. To overcome these issues, we propose two innovative strategies: the mixed task-guided feature enhancement, which directs feature extraction toward lesion-related details across scales, and the prompt-guided detail feature completion, which integrates coarse- and fine-grained features from WSI based on specific prompts without compromising inference speed. Leveraging a comprehensive dataset of 490K samples from diverse pathology tasks, we trained the pathology-specialized LVLM, OmniPath. Extensive experiments demonstrate that this model significantly outperforms existing methods in diagnostic accuracy and efficiency, providing an interactive, clinically aligned approach for auxiliary diagnosis in a wide range of pathology applications.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 61 references that are in the map. First posted on arXiv 12 Dec 2024.
Builds on
38 of 61 references.
Closest ancestors · 4
Same direction, newest first
- 2024PathAlign: A vision-language model for whole slide images in histopathology27 Jun
- 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology29 Jan
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
- 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May
Other references · 34
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Segment Anythingcited by 8,158
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
22 more
- 2018Attention-based Deep Multiple Instance Learningcited by 1,077
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
- 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
- 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
- 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
- 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
- 2020PanNuke Dataset Extension, Insights and Baselinescited by 113
- 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
- 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
- 2019CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentationcited by 73
- 2021A Petri Dish for Histopathology Image Analysiscited by 63
- 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
- 2019Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networkscited by 43
- 2022Differentiable Zooming for Multiple Instance Learning on Whole-Slide Imagescited by 43
- 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
- 2021NuCLS: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentationcited by 33
- 2023CoNIC Challenge: Pushing the Frontiers of Nuclear Detection, Segmentation, Classification and Countingcited by 33
- 2023NuInsSeg: A Fully Annotated Dataset for Nuclei Instance Segmentation in H&E-Stained Histological Imagescited by 29
- 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 15
- 2024PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathologycited by 6
- 2021Edge-competing Pathological Liver Vessel Segmentation with Limited Labelscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
- 2024WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image
- 2025A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
- 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
- 2025LanGuSTE: Language-Guided Coarse-to-Fine Patch Selection for Efficient Whole Slide Image Analysis
- 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models
- 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
- 2024Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement