mlsanityData to 7 Oct 2026

Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning

arXiv 2412.08614 ↗First posted 11 Dec 2024cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 59 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

41 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2024GPT-4o System Cardcited by 4,467
  12. 2022A ConvNet for the 2020scited by 3,545
28 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  13. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  14. 2018nocaps: novel object captioning at scalecited by 444
  15. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  16. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  17. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  18. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  19. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  20. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  21. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  22. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  23. 2024Benchmarking and Improving Detail Image Captioncited by 81
  24. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  25. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  26. 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
  27. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  28. 2023Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 16.

30 papers.

Sort:

By area:

  1. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  2. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  3. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
  4. 2025SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generationcited by 7
  5. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  6. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  7. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  8. 2025RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstructioncited by 4
  9. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  10. 2025Harnessing Caption Detailness for Data-Efficient Text-to-Image Generationcited by 3
  11. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  12. 2025Multigranular Evaluation for Brain Visual Decodingcited by 2
18 more
  1. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  2. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  3. 2026IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memorycited by 2
  4. 2025MPCAR: Multi-Perspective Contextual Augmentation for Enhanced Visual Reasoning in Large Vision-Language Modelscited by 1
  5. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  6. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  7. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
  8. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  9. 2026CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoTcited by 0
  10. 2026CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuningcited by 0
  11. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  12. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 0
  13. 2026CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answeringcited by 0
  14. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  15. 2026Controllable Image Captioning with Prompt-Conditioned Scene Rewardscited by 0
  16. 2026PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectioncited by 0
  17. 2026Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understandingcited by 0
  18. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
  2. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
  3. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
  4. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
  5. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
  6. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  7. 2023CapsFusion: Rethinking Image-Text Data at Scale
  8. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding