mlsanityData to 7 Oct 2026

LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model

arXiv 2401.02330 ↗First posted 4 Jan 2024cs.CV cs.CLcited by 128

Not on the map. Why?

Cited by 73 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce LLaVA-φ (LLaVA-Phi), an efficient multi-modal assistant that harnesses the power of the recently advanced small language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a notable advancement in the realm of compact multi-modal models. It demonstrates that even smaller language models, with as few as 2.7B parameters, can effectively engage in intricate dialogues that integrate both textual and visual elements, provided they are trained with high-quality corpora. Our model delivers commendable performance on publicly available benchmarks that encompass visual comprehension, reasoning, and knowledge-based perception. Beyond its remarkable performance in multi-modal dialogue tasks, our model opens new avenues for applications in time-sensitive environments and systems that require real-time interaction, such as embodied agents. It highlights the potential of smaller language models to achieve sophisticated levels of understanding and interaction, while maintaining greater resource efficiency.The project is available at https://github.com/zhuyiche/llava-phi.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

33 of 44 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
21 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2023Textbooks Are All You Needcited by 561
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  15. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  16. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  17. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  18. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  19. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  20. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  21. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspectivecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 74; 2025: 38; 2026: 16.

128 papers.

Sort:

By area:

  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  5. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  7. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  9. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  10. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  11. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  12. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
38 more
  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  3. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  4. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  5. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  6. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  7. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  8. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  9. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  10. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  11. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  12. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  13. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  14. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  15. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  16. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  17. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  18. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  19. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  20. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  21. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  22. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  23. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  24. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  25. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 37
  26. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  27. 2024AutoPSV: Automated Process-Supervised Verifiercited by 28
  28. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  29. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  30. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  31. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
  32. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  33. 2024The Evolution of Multimodal Model Architecturescited by 21
  34. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  35. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  36. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  37. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  38. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19

50 most cited of 128 All 128 (JSON)