mlsanityData to 7 Oct 2026

COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation

arXiv 2502.02589 ↗First posted 4 Feb 2025cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper introduces the COCONut-PanCap dataset, created to enhance panoptic segmentation and grounded image captioning. Building upon the COCO dataset with advanced COCONut panoptic masks, this dataset aims to overcome limitations in existing image-text datasets that often lack detailed, scene-comprehensive descriptions. The COCONut-PanCap dataset incorporates fine-grained, region-level captions grounded in panoptic segmentation masks, ensuring consistency and improving the detail of generated captions. Through human-edited, densely annotated descriptions, COCONut-PanCap supports improved training of vision-language models (VLMs) for image understanding and generative models for text-to-image tasks. Experimental results demonstrate that COCONut-PanCap significantly boosts performance across understanding and generation tasks, offering complementary benefits to large-scale datasets. This dataset sets a new benchmark for evaluating models on joint panoptic segmentation and grounded captioning tasks, addressing the need for high-quality, detailed image-text annotations in multi-modal learning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

52 of 72 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  19. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  20. 2018Panoptic Segmentationcited by 796
  21. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  22. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  23. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  25. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  27. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  28. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  29. 2023PACO: Parts and Attributes of Common Objectscited by 146
  30. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  31. 2022Panoptic Scene Graph Generationcited by 114
  32. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  33. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  34. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  35. 2024Benchmarking and Improving Detail Image Captioncited by 81
  36. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  37. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  38. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025FlowTok: Flowing Seamlessly Across Text and Image Tokenscited by 31
  2. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 19
  3. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  4. 2025Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformerscited by 13
  5. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  6. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methodscited by 5
  7. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  8. 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Designcited by 3
  9. 2026Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervisioncited by 2
  10. 2025MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suitescited by 1
  11. 2026Vision-Language Grounding as Bidirectional Concept Correspondencecited by 0
  12. 2026PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning
  2. 2025Panoptic Captioning: An Equivalence Bridge for Image and Text
  3. 2026MIRCaps: A Large-Scale Mixed-Domain Dataset with Image-Level and Region-Level Captions for Fine-Grained Vision-Language Learning
  4. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  5. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity
  6. 2024BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs
  7. 2025Discovering Meaningful Units with Visually Grounded Semantics from Image Captions
  8. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning