mlsanityData to 7 Oct 2026

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

arXiv 2312.16862 ↗First posted 28 Dec 2023cs.CV cs.CLcited by 77

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In recent years, multimodal large language models (MLLMs) such as GPT-4V have demonstrated remarkable advancements, excelling in a variety of vision-language tasks. Despite their prowess, the closed-source nature and computational demands of such models limit their accessibility and applicability. This study introduces TinyGPT-V, a novel open-source MLLM, designed for efficient training and inference across various vision-language tasks, including image captioning (IC) and visual question answering (VQA). Leveraging a compact yet powerful architecture, TinyGPT-V integrates the Phi-2 language model with pre-trained vision encoders, utilizing a unique mapping module for visual and linguistic information fusion. With a training regimen optimized for small backbones and employing a diverse dataset amalgam, TinyGPT-V requires significantly lower computational resources 24GB for training and as little as 8GB for inference without compromising on performance. Our experiments demonstrate that TinyGPT-V, with its language model 2.8 billion parameters, achieves comparable results in VQA and image inference tasks to its larger counterparts while being uniquely suited for deployment on resource-constrained devices through innovative quantization techniques. This work not only paves the way for more accessible and efficient MLLMs but also underscores the potential of smaller, optimized models in bridging the gap between high performance and computational efficiency in real-world applications. Additionally, this paper introduces a new approach to multimodal large language models using smaller backbones. Our code and training weights are available in the supplementary material.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 50 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

24 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  2. 2023ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter12 May
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
9 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  8. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  9. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 30; 2026: 8.

77 papers.

Sort:

By area:

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  6. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  7. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  8. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  11. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 120
  12. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
38 more
  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  3. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  5. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  6. 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perceptioncited by 53
  7. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  8. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  9. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  10. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  11. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  12. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layercited by 27
  13. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  14. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  15. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  16. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  17. 2024The Evolution of Multimodal Model Architecturescited by 21
  18. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  19. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  20. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  21. 2025Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Modelscited by 16
  22. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  23. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  24. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  25. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  26. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  27. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Expertscited by 11
  28. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  29. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 10
  30. 2024Efficient Multi-modal Large Language Models via Visual Token Groupingcited by 9
  31. 2024SLaVA-CXR: Small Language and Vision Assistant for Chest X-ray Report Automationcited by 7
  32. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  33. 2025PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Modelscited by 6
  34. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  35. 2025SEMPO: Lightweight Foundation Models for Time Series Forecastingcited by 6
  36. 2025Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networkscited by 5
  37. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  38. 2024Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6Gcited by 4

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
  2. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  3. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  4. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
  5. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
  6. 2025Benchmarking Large and Small MLLMs
  7. 2025SmolVLM: Redefining small and efficient multimodal models
  8. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation