mlsanityData to 14 Oct 2026

Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

arXiv 2412.05939 ↗First posted 8 Dec 2024cs.CV cs.CL cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions). We hypothesize that integrating fine-grained concept annotations (e.g., object labels and object regions) will further improve performance, as both data granularities complement each other in terms of breadth and depth in concept representation. We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we explore the impact of different data recipes on multimodal comprehension and generation. Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and a general MLLM framework. We clearly explore and demonstrate the potential of MMGiC to help MLLMs better locate and learn concepts, aligning vision and language at multiple granularities. We further validate our hypothesis by investigating the fair comparison and effective collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks, e.g., their appropriate combination achieve 3.95% and 2.34% absolute improvements over image--caption data alone on POPE and SEED-Bench. Code, data and models will be available at https://github.com/LooperXX/MMGiC.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 135 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

42 of 135 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  2. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  3. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception11 Jul
  4. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens17 Jun
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  6. 2023Generative Multimodal Models are In-Context Learners20 Dec
  7. 2023VILA: On Pre-training for Visual Language Models12 Dec
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models4 Oct
  12. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants1 Oct
4 more
  1. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
14 more
  1. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  2. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  3. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  4. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  5. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  6. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  7. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  8. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  9. 2024Large Language Models Meet NLP: A Surveycited by 103
  10. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  11. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  12. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  13. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32
  14. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  2. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  4. 2024MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
  5. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  6. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey