mlsanityData to 14 Oct 2026

ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

arXiv 2402.11684 ↗First posted 18 Feb 2024cs.CL cs.AIcited by 205

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset textitALLaVA, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

Key facts. Cited by 205 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 44 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

20 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
7 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  2. 2018nocaps: novel object captioning at scalecited by 444
  3. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  4. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  5. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  6. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  7. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 94; 2025: 85; 2026: 26.

205 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  6. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  7. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  8. 2024OmniGen: Unified Image Generationcited by 394
  9. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  10. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  11. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  12. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
38 more
  1. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  2. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  3. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  4. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  5. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  7. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  10. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  11. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  12. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  13. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  14. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  15. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  16. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
  17. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  18. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  19. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  20. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  21. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  22. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  23. 2024Benchmarking and Improving Detail Image Captioncited by 81
  24. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  25. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  26. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  27. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  28. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  29. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  30. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  31. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  32. 2024Reconstructive Visual Instruction Tuningcited by 55
  33. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  34. 2025FineVision: Open Data Is All You Needcited by 53
  35. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  36. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  37. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  38. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45

50 most cited of 205 All 205 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  2. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  3. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
  4. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  8. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models