mlsanityData to 14 Oct 2026

On Domain-Adaptive Post-Training for Multimodal Large Language Models

arXiv 2411.19930 ↗First posted 29 Nov 2024cs.CL cs.CV cs.LGcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Adapting general multimodal large language models (MLLMs) to specific domains, such as scientific and industrial fields, is highly significant in promoting their practical applications. This paper systematically investigates domain adaptation of MLLMs via post-training, focusing on data synthesis, training pipeline, and task evaluation. (1) Data Synthesis: Using only open-source models, we develop a generate-then-filter pipeline that curates diverse visual instruction tasks based on domain-specific image-caption pairs. The resulting data surpass the data synthesized by manual rules or strong closed-source models in enhancing domain-specific performance. (2) Training Pipeline: Unlike general MLLMs that typically adopt a two-stage training paradigm, we find that a single-stage approach is more effective for domain adaptation. (3) Task Evaluation: We conduct extensive experiments in high-impact domains such as biomedicine, food, and remote sensing, by post-training a variety of MLLMs and then evaluating MLLM performance on various domain-specific tasks. Finally, we fully open-source our models, code, and data to encourage future research in this area.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 59 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

34 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  3. 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Model22 Dec
  4. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator11 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  10. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  11. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
17 more
  1. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  2. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  3. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 207
  4. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  5. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  6. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 184
  7. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  8. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  9. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  10. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  11. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 100
  12. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  13. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  14. 2021Nutrition5k: Towards Automatic Nutritional Understanding of Generic Foodcited by 58
  15. 2021A Large-Scale Benchmark for Food Image Segmentationcited by 45
  16. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  17. 2024LLaVA-Chef: A Multi-modal Generative Model for Food Recipescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 8.

13 papers.

Sort:

By area:

  1. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  2. 2025OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learningcited by 24
  3. 2025Calibrating Uncertainty Quantification of Multi-Modal LLMs using Groundingcited by 8
  4. 2026AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparisoncited by 5
  5. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  6. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  7. 2025GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMscited by 1
  8. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuningcited by 1
  9. 2026CharTool: Tool-Integrated Visual Reasoning for Chart Understandingcited by 0
  10. 2026FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Imagescited by 0
  11. 2026FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysiscited by 0
  12. 2026Evaluating and Understanding Model Editing for Medical Vision Language Modelscited by 0
1 more
  1. 2026How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  3. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  4. 2024Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning
  5. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  6. 2024MileBench: Benchmarking MLLMs in Long Context
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models