mlsanityData to 14 Oct 2026

MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants

arXiv 2412.12661 ↗First posted 17 Dec 2024cs.AI cs.CL cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent advancements in mixed-modal generative have opened new avenues for developing unified biomedical assistants capable of analyzing biomedical images, answering complex questions about them, and generating multimodal patient reports. However, existing datasets face challenges such as small sizes, limited coverage of biomedical tasks and domains, and a reliance on narrow sources. To address these gaps, we present MedMax, a large-scale multimodal biomedical instruction-tuning dataset for mixed-modal foundation models. With 1.47 million instances, MedMax encompasses a diverse range of tasks, including interleaved image-text generation, biomedical image captioning and generation, visual chat, and report understanding. These tasks span knowledge across diverse biomedical domains, including radiology and histopathology, grounded in medical papers and YouTube videos. Subsequently, we fine-tune a mixed-modal foundation model on the MedMax dataset, achieving significant performance improvements: a 26% gain over the Chameleon model and an 18.3% improvement over GPT-4o across 12 downstream biomedical visual question-answering tasks. Finally, we introduce a unified evaluation suite for biomedical tasks to guide the development of mixed-modal biomedical AI assistants. The data, model, and code is available at https://mint-medmax.github.io/.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 55 of its 72 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

55 of 72 references.

References · 55

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  7. 2021The Medical Segmentation Decathloncited by 628
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  10. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  12. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  13. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  14. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  15. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  16. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  17. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  18. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  19. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 309
  20. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  21. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  23. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
  24. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  25. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  26. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  27. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  28. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  30. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  31. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  32. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  33. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  34. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
  35. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  36. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  37. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 81
  38. 2024MonoFormer: One Transformer for Both Diffusion and Autoregressioncited by 46

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
  2. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  3. 2025Mediator-Guided Multi-Agent Collaboration among Open-Source Models for Medical Decision-Makingcited by 5
  4. 2025EndoCogniAgent: Closed-Loop Agentic Reasoning with Self-Consistency Validation for Endoscopic Diagnosiscited by 3
  5. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRIcited by 2
  6. 2026Aloe-Vision: Robust Vision-Language Models for Healthcarecited by 1