mlsanityData to 7 Oct 2026

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

arXiv 2405.09215 ↗First posted 15 May 2024cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Xmodel-VLM, a cutting-edge multimodal vision language model. It is designed for efficient deployment on consumer GPU servers. Our work directly confronts a pivotal industry issue by grappling with the prohibitive service costs that hinder the broad adoption of large-scale multimodal systems. Through rigorous training, we have developed a 1B-scale language model from the ground up, employing the LLaVA paradigm for modal alignment. The result, which we call Xmodel-VLM, is a lightweight yet powerful multimodal vision language model. Extensive testing across numerous classic multimodal benchmarks has revealed that despite its smaller size and faster execution, Xmodel-VLM delivers performance comparable to that of larger models. Our model checkpoints and code are publicly available on GitHub at https://github.com/XiaoduoAILab/XmodelVLM.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

35 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
15 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  13. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  14. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  15. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2025A Survey on Efficient Vision-Language Modelscited by 19
  3. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  4. 2025Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchucited by 5
  5. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  3. 2025Mordal: Automated Pretrained Model Selection for Vision Language Models
  4. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  8. 2024An Introduction to Vision-Language Modeling