mlsanityData to 14 Oct 2026

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology

arXiv 2607.14581 ↗First posted 16 Jul 2026cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advances in large language models (LLMs) and their extension to vision-language models (VLMs) have made it easier to combine text and images for tasks such as report generation. Existing VLMs in medicine typically focus on 2D images (chest X-rays), and their extension to 3D imaging has been difficult because of the lack of paired 3D imaging-text data. Thus, we introduce a new method for creating a 3D image-text dataset for brain oncology using 3D MRI scans of glioma and meningioma cases. We use a cooperative system in which several LLMs work together to generate and check reports, ensuring that they are accurate and clear. By leveraging the new 3D MRI-text dataset, we further build a VLM that converts MRI scans into tokens and aligns them with text instructions. Our VLM performed better in report generation and visual question answering tasks than other 2D and 3D methods. Our method not only improves the quality of reports but also helps with better diagnosis and treatment in brain oncology.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 39 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

23 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model18 Oct
  2. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  3. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 518
7 more
  1. 2023Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Studycited by 64
  2. 2023The ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2023: Intracranial Meningiomacited by 53
  3. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  4. 2023Vision-Language Generative Model for View-Specific Chest X-ray Generationcited by 14
  5. 2024A Novel Approach to Eliminating Hallucinations in Large Language Model-Assisted Causal Discoverycited by 1
  6. 2025Privacy-Preserving Chest X-ray Classification in Latent Space with Homomorphically Encrypted Neural Inferencecited by 1
  7. 2026Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
  2. 2026Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
  3. 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
  4. 2026Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
  5. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
  6. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
  7. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease
  8. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI