A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Accurate PSMA PET/CT interpretation is central to prostate cancer management, yet existing PET/CT AI models typically address isolated tasks. We propose a unified PSMA PET/CT vision-language model for report generation, visual question answering, and lesion segmentation. The framework adopts an LLaVA-style architecture, comprising a PET/CT vision encoder, an MLP-Mixer projection module, a LoRA-tuned large language model, and a 3D segmentation branch. Training followed a four-stage strategy: vision encoder pretraining, projection-layer alignment, VLM fine-tuning, and final multitask tuning. Language tasks used 5,747 PSMA PET/CT datasets with paired reports, while segmentation used the PSMA subset of AutoPET. The model outperformed PET2REP and a CT-based baseline across standard report-generation metrics, improved performance across VQA question types, and achieved higher Dice and lesion-level overlap F1 than SegAnyPET and nnUNet. These results support the feasibility of a unified framework for structured, interactive, interpretable PSMA PET/CT analysis with voxel-level grounding within a single multitask model architecture.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 41 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
13 of 41 references.
Closest ancestors · 3
Same direction, newest first
- 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation14 Jan
- 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis25 Mar
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
Other references · 10
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
- 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
- 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
- 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
- 2025SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Imagescited by 11
- 2025PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomographycited by 7
- 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
- 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
- 2025Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings
- 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation
- 2026An end-to-end-trained vision-language model for native-language prostate pathology report generation
- 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
- 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
- 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering