mlsanityData to 7 Oct 2026

Panoptic Narrative Grounding

arXiv 2109.04988 ↗First posted 10 Sep 2021cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as stepping stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level by using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. To guarantee the quality of our annotations, we take advantage of the semantic structure contained in WordNet to exclusively incorporate noun phrases that are grounded to a meaningfully related panoptic segmentation region. The proposed baseline achieves a performance of 55.4 absolute Average Recall points. This result is a suitable foundation to push the envelope further in the development of methods for Panoptic Narrative Grounding.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

31 of 53 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
19 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2018Panoptic Segmentationcited by 796
  3. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  4. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  6. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  7. 2016Segmentation from Natural Language Expressionscited by 257
  8. 2019Connecting Vision and Language with Localized Narrativescited by 225
  9. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  10. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  11. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  12. 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 127
  13. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  14. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  15. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  16. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  17. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  18. 2020Dual Convolutional LSTM Network for Referring Image Segmentationcited by 12
  19. 2020Cross-Modality Relevance for Reasoning on Language and Visioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 6; 2025: 8; 2026: 3.

27 papers.

Sort:

By area:

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  2. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  3. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  4. 2023Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentationcited by 28
  5. 2022Detailed Annotations of Chest X-Rays via CT Projection for Report Understandingcited by 24
  6. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  7. 2025Multimodal Referring Segmentation: A Surveycited by 22
  8. 2023Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Networkcited by 13
  9. 2024High-Quality Entity Segmentation and Groundingcited by 13
  10. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  11. 2022PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Groundingcited by 12
  12. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
15 more
  1. 2023NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learningcited by 8
  2. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 8
  3. 2023Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Groundingcited by 6
  4. 2023Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Networkcited by 5
  5. 2024Point-VOS: Pointing Up Video Object Segmentationcited by 5
  6. 2023Semi-Supervised Panoptic Narrative Groundingcited by 4
  7. 2024Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Groundingcited by 4
  8. 2025GroundCap: A Visually Grounded Image Captioning Datasetcited by 4
  9. 2025MedicalNarratives: Connecting Medical Vision and Language with Localized Narrativescited by 2
  10. 2025GS: Generative Segmentation via Label Diffusioncited by 2
  11. 2022Fantastic Data and How to Query Themcited by 1
  12. 2025REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decodingcited by 1
  13. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1
  14. 20263D-DRES: Detailed 3D Referring Expression Segmentationcited by 1
  15. 2026PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectioncited by 0