mlsanityData to 14 Oct 2026

Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

arXiv 2411.09691 ↗First posted 14 Nov 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multi-modal large language models (MLLMs) have achieved remarkable success in fine-grained visual understanding across a range of tasks. However, they often encounter significant challenges due to inadequate alignment for fine-grained knowledge, which restricts their ability to accurately capture local details and attain a comprehensive global perception. While recent advancements have focused on aligning object expressions with grounding information, they typically lack explicit integration of object images, which contain affluent information beyond mere texts or coordinates. To bridge this gap, we introduce a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. This innovative method is underpinned by our multi-scale fine-grained enhancement data synthesis pipeline, which provides over 300K essential training data to enhance alignment and improve overall performance. Furthermore, we present TinyGroundingGPT, a series of compact models optimized for high-level alignments. With a scale of approximately 3B parameters, TinyGroundingGPT achieves outstanding results in grounding tasks while delivering performance comparable to larger MLLMs in complex visual scenarios.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

48 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  2. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  3. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Model11 Jan
  4. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects8 Dec
  5. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  6. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  7. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  10. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
26 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  13. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  14. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  16. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  17. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  18. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  19. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  20. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  21. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  22. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  23. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 71
  24. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  25. 2024UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Modelcited by 13
  26. 2024QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12
  2. 2025EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editingcited by 12
  3. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  4. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Modelscited by 9
  5. 2026MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledgecited by 5
  6. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Modelscited by 4
  7. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 4
  8. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  9. 2025Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Modelscited by 0
  10. 2026FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuningcited by 0
  11. 2026Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
  2. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  3. 2025GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
  5. 2025Visual Representation Alignment for Multimodal Large Language Models
  6. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  7. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding
  8. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models