mlsanityData to 14 Oct 2026

OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects

arXiv 2410.01261 ↗First posted 2 Oct 2024cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

There is a gap in the understanding of occluded objects in existing large-scale visual language multi-modal models. Current state-of-the-art multimodal models fail to provide satisfactory results in describing occluded objects for visual-language multimodal models through universal visual encoders. Another challenge is the limited number of datasets containing image-text pairs with a large number of occluded objects. Therefore, we introduce a novel multimodal model that applies a newly designed visual encoder to understand occluded objects in RGB images. We also introduce a large-scale visual-language pair dataset for training large-scale visual-language multimodal models and understanding occluded objects. We start our experiments comparing with the state-of-the-art models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 17 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

14 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  2. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 11

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  9. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  10. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  11. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 50

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Countingcited by 38
  2. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 3
  3. 2025Object Counting with GPT-4o and GPT-5: A Comparative Studycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learning
  2. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model
  3. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  4. 2024E5-V: Universal Embeddings with Multimodal Large Language Models
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  8. 2023MLLMs-Augmented Visual-Language Representation Learning