mlsanityData to 14 Oct 2026

TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models

arXiv 2405.11788 ↗First posted 20 May 2024cs.LGcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present TinyLLaVA Factory, an open-source modular codebase for small-scale large multimodal models (LMMs) with a focus on simplicity of code implementations, extensibility of new features, and reproducibility of training results. Following the design philosophy of the factory pattern in software engineering, TinyLLaVA Factory modularizes the entire system into interchangeable components, with each component integrating a suite of cutting-edge models and methods, meanwhile leaving room for extensions to more features. In addition to allowing users to customize their own LMMs, TinyLLaVA Factory provides popular training recipes to let users pretrain and finetune their models with less coding effort. Empirical experiments validate the effectiveness of our codebase. The goal of TinyLLaVA Factory is to assist researchers and practitioners in exploring the wide landscape of designing and training small-scale LMMs with affordable computational resources.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 32 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

25 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
10 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  8. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  2. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  3. 2025MedM-VL: What Makes a Good Medical LVLM?cited by 4
  4. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  5. 2025RLLaVA: An RL-central Framework for Language and Vision Assistantscited by 1
  6. 2024TinyEmo: Scaling down Emotional Reasoning via Metric Projectioncited by 0
  7. 2025Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Multimodal Models for Low-Resource Languages: A Survey
  2. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  3. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
  4. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  5. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devices
  6. 2024Automatic benchmarking of large multimodal models via iterative experiment programming
  7. 2026From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning
  8. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training