# PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding

arXiv: https://arxiv.org/abs/2208.05647 · HTML: https://mlsanity.com/paper/2208.05647/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 83 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)

## Built on by (most cited first, 12 of 12)

- [LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding](https://mlsanity.com/paper/2501.08282/) (2025)
- [3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2308.16632/) (2023)
- [High-Quality Entity Segmentation and Grounding](https://mlsanity.com/paper/2402.02555/) (2024)
- [Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model](https://mlsanity.com/paper/2407.05352/) (2024)
- [NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning](https://mlsanity.com/paper/2310.10975/) (2023)
- [Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding](https://mlsanity.com/paper/2311.01091/) (2023)
- [Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Network](https://mlsanity.com/paper/2310.16616/) (2023)
- [Semi-Supervised Panoptic Narrative Grounding](https://mlsanity.com/paper/2310.18142/) (2023)
- [Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding](https://mlsanity.com/paper/2409.08251/) (2024)
- [Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision](https://mlsanity.com/paper/2504.21423/) (2025)
- [GS: Generative Segmentation via Label Diffusion](https://mlsanity.com/paper/2508.20020/) (2025)
- [3D-DRES: Detailed 3D Referring Expression Segmentation](https://mlsanity.com/paper/2603.02896/) (2026)

## Abstract (arXiv)

Panoptic Narrative Grounding (PNG) is an emerging task whose goal is to segment visual objects of things and stuff categories described by dense narrative captions of a still image. The previous two-stage approach first extracts segmentation region proposals by an off-the-shelf panoptic segmentation model, then conducts coarse region-phrase matching to ground the candidate regions for each noun phrase. However, the two-stage pipeline usually suffers from the performance limitation of low-quality proposals in the first stage and the loss of spatial details caused by region feature pooling, as well as complicated strategies designed for things and stuff categories separately. To alleviate these drawbacks, we propose a one-stage end-to-end Pixel-Phrase Matching Network (PPMN), which directly matches each phrase to its corresponding pixels instead of region proposals and outputs panoptic segmentation by simple combination. Thus, our model can exploit sufficient and finer cross-modal semantic correspondence from the supervision of densely annotated pixel-phrase pairs rather than sparse region-phrase pairs. In addition, we also propose a Language-Compatible Pixel Aggregation (LCPA) module to further enhance the discriminative ability of phrase features through multi-round refinement, which selects the most compatible pixels for each phrase to adaptively aggregate the corresponding visual context. Extensive experiments show that our method achieves new state-of-the-art performance on the PNG benchmark with 4.0 absolute Average Recall gains.