mlsanityData to 14 Oct 2026

MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output

arXiv 2510.10011 ↗First posted 11 Oct 2025cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Currently, medical vision language models are widely used in medical vision question answering tasks. However, existing models are confronted with two issues: for input, the model only relies on text instructions and lacks direct understanding of visual clues in the image; for output, the model only gives text answers and lacks connection with key areas in the image. To address these issues, we propose a unified medical vision language model MIMO, with visual referring Multimodal Input and pixel grounding Multimodal Output. MIMO can not only combine visual clues and textual instructions to understand complex medical images and semantics, but can also ground medical terminologies in textual output within the image. To overcome the scarcity of relevant data in the medical field, we propose MIMOSeg, a comprehensive medical multimodal dataset including 895K samples. MIMOSeg is constructed from four different perspectives, covering basic instruction following and complex question answering with multimodal input and multimodal output. We conduct experiments on several downstream medical multimodal tasks. Extensive experimental results verify that MIMO can uniquely combine visual referring and pixel grounding capabilities, which are not available in previous models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 99 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

66 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Segment Anythingcited by 8,234
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
38 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  2. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  3. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  8. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  9. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 518
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  11. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  13. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  14. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  15. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 373
  16. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  17. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  18. 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 224
  19. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  20. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 198
  21. 2019The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challengecited by 190
  22. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  23. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  24. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  25. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  26. 2020Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domainscited by 124
  27. 2023SAM-Med2Dcited by 124
  28. 2020VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Imagescited by 111
  29. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 106
  30. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  31. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  32. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  33. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  34. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  35. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  36. 2018Grounded Video Descriptioncited by 78
  37. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  38. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2026MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Datacited by 5
  2. 2025vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMscited by 3
  3. 2025Parameter Importance-Driven Continual Learning for Foundation Modelscited by 2
  4. 2025Foundation Models in Biomedical Imaging: Turning Hype into Realitycited by 2
  5. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  6. 2026Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidencecited by 1
  7. 2026Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMscited by 1
  8. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQAcited by 1
  9. 2026Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoningcited by 0
  10. 2026CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inferencecited by 0
  11. 2026EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Modelscited by 0
  12. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Modelscited by 0
3 more
  1. 2026Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosiscited by 0
  2. 2026LocAnyMed: Vision-Language Grounding for Multimodal Medical Imagescited by 0
  3. 2026Scaling 3D Visual Grounding in Abdominal CTcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
  2. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  3. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  4. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
  5. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  6. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation
  7. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
  8. 2024Targeted Visual Prompting for Medical Visual Question Answering