mlsanityData to 7 Oct 2026

FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity

arXiv 2411.15411 ↗First posted 23 Nov 2024cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The advent of large Vision-Language Models (VLMs) has significantly advanced multimodal tasks, enabling more sophisticated and accurate reasoning across various applications, including image and video captioning, visual question answering, and cross-modal retrieval. Despite their superior capabilities, VLMs struggle with fine-grained image regional composition information perception. Specifically, they have difficulty accurately aligning the segmentation masks with the corresponding semantics and precisely describing the compositional aspects of the referred regions. However, compositionality - the ability to understand and generate novel combinations of known visual and textual components - is critical for facilitating coherent reasoning and understanding across modalities by VLMs. To address this issue, we propose FINECAPTION, a novel VLM that can recognize arbitrary masks as referential inputs and process high-resolution images for compositional image captioning at different granularity levels. To support this endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region compositional image captioning, which introduces the task of compositional attribute-aware regional image captioning. Empirical results demonstrate the effectiveness of our proposed model compared to other state-of-the-art VLMs. Additionally, we analyze the capabilities of current VLMs in recognizing various visual prompts for compositional region image captioning, highlighting areas for improvement in VLM design and training.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 55 references that are in the map. First posted on arXiv 23 Nov 2024.

Builds on

47 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models13 Oct

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2022A ConvNet for the 2020scited by 3,545
34 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2016Modeling Context in Referring Expressionscited by 1,054
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  11. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  13. 2016Image Captioning with Semantic Attentioncited by 260
  14. 2024Generative Representational Instruction Tuningcited by 227
  15. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  16. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  17. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  20. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  21. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  22. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  23. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  24. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  25. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  26. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  27. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  28. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  29. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  30. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  31. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  32. 2024ControlCap: Controllable Region-level Captioningcited by 13
  33. 2022Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularizationcited by 12
  34. 2024Chain-of-Thought Prompting for Demographic Inference with Large Multimodal Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  2. 2025Generative AI for Cel-Animation: A Surveycited by 17
  3. 2025Top-Down Semantic Refinement for Image Captioningcited by 16
  4. 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 15
  5. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Promptingcited by 9
  6. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  7. 2026On the Generalization Capacities of MLLMs for Spatial Intelligencecited by 7
  8. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  9. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 6
  10. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  11. 2025ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restorationcited by 5
  12. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
14 more
  1. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  2. 2025UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understandingcited by 4
  3. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  4. 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 3
  5. 2025URECA: Unique Region Caption Anythingcited by 2
  6. 2025One Patch to Caption Them All: A Unified Zero-Shot Captioning Frameworkcited by 2
  7. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  8. 2025Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captionscited by 1
  9. 2026Indexing Multimodal Language Models for Large-scale Image Retrievalcited by 1
  10. 2025CAS-IQA: Teaching Vision-Language Models for Synthetic Angiography Quality Assessmentcited by 0
  11. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interactioncited by 0
  12. 2026PortraitCraft: A Benchmark for Portrait Composition Understanding and Generationcited by 0
  13. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  14. 2026EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
  2. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  3. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
  4. 2024ComAlign: Compositional Alignment in Vision-Language Models
  5. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  6. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  7. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception
  8. 2023Text encoders bottleneck compositionality in contrastive vision-language models