mlsanityData to 7 Oct 2026

TinyLLaVA: A Framework of Small-scale Large Multimodal Models

arXiv 2402.14289 ↗First posted 22 Feb 2024cs.LG cs.CLcited by 164

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 63 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

30 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
9 more
  1. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  6. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  7. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  8. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  9. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 63; 2025: 66; 2026: 35.

164 papers.

Sort:

By area:

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  4. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  5. 2024What matters when building vision-language models?cited by 321
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  7. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  8. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  9. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  12. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
38 more
  1. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  6. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  7. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  8. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  9. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  10. 2024Dense Connector for MLLMscited by 48
  11. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  12. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  13. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  14. 2024Automated Multi-level Preference for MLLMscited by 29
  15. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  16. 2024UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarioscited by 28
  17. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layercited by 27
  18. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Modelscited by 27
  19. 2024OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Designcited by 26
  20. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  21. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  22. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  23. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  24. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  25. 2025A Survey on Efficient Vision-Language Modelscited by 19
  26. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 18
  27. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  28. 2025UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learningcited by 16
  29. 2025Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correctioncited by 16
  30. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  31. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  32. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  33. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  34. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 14
  35. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  36. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
  37. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  38. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13

50 most cited of 164 All 164 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
  2. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  5. 2025TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler
  6. 2025Large Multimodal Models for Low-Resource Languages: A Survey
  7. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  8. 2025Scaling Laws for Native Multimodal Models