mlsanityData to 7 Oct 2026

Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

arXiv 2403.06199 ↗First posted 10 Mar 2024cs.CV cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/llava-phi.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 65 references that are in the map. First posted on arXiv 10 Mar 2024.

Builds on

55 of 65 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
2 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
29 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  13. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  14. 2019Towards VQA Models That Can Readcited by 1,696
  15. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  19. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  20. 2021ImageNet-21K Pretraining for the Massescited by 607
  21. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  22. 2023Textbooks Are All You Needcited by 561
  23. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  24. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  25. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  26. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  27. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  28. 2013Stabilitycited by 119
  29. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 2.

24 papers.

Sort:

By area:

  1. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  2. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  3. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  4. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  5. 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuningcited by 23
  6. 2025Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Frameworkcited by 23
  7. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  8. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  9. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  10. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  11. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  12. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
12 more
  1. 2025DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergycited by 13
  2. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  3. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  4. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  5. 2024Olympus: A Universal Task Router for Computer Vision Taskscited by 4
  6. 2025Towards Fast, Memory-based and Data-Efficient Vision-Language Policycited by 2
  7. 2025Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Studycited by 1
  8. 2025Multimodal Chip Physical Design Engineer Assistantcited by 1
  9. 2026Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understandingcited by 1
  10. 2025LOP: Learning Optimal Pruning for Efficient On-Demand MLLMs Scalingcited by 0
  11. 2025Improving MLLM Training Efficiency via Stage-Aware Sparsitycited by 0
  12. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2024Efficient Multimodal Large Language Models: A Survey
  3. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2024A Survey on Benchmarks of Multimodal Large Language Models
  6. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  7. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  8. 2025Benchmarking Large and Small MLLMs