mlsanityData to 7 Oct 2026

PolyFormer: Referring Image Segmentation as Sequential Polygon Generation

arXiv 2302.07387 ↗First posted 14 Feb 2023cs.CVcited by 147

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any coordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging RefCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J&F on the Ref-DAVIS17 dataset.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Builds on 77 of its 95 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

77 of 95 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2017Mask R-CNNcited by 7,259
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
38 more
  1. 2016Wide Residual Networkscited by 3,057
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2015Fast R-CNNcited by 2,930
  4. 2018YOLOv3: An Incremental Improvementcited by 2,652
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Spatial Transformer Networkscited by 1,423
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  12. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  13. 2019Objects as Pointscited by 1,172
  14. 2016Modeling Context in Referring Expressionscited by 1,054
  15. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  16. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  17. 2018Path Aggregation Network for Instance Segmentationcited by 796
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  20. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  21. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  22. 2019YOLACT: Real-time Instance Segmentationcited by 441
  23. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  24. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  25. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  26. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  27. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  30. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  31. 2016Segmentation from Natural Language Expressionscited by 257
  32. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  33. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  34. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  35. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  36. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  37. 2019PolarMask: Single Shot Instance Segmentation with Polar Representationcited by 188
  38. 2022Language as Queries for Referring Video Object Segmentationcited by 180

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 50; 2025: 42; 2026: 24.

147 papers.

Sort:

By area:

  1. 2023Segment Everything Everywhere All at Oncecited by 546
  2. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  3. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  4. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  5. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  6. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  7. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  8. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  9. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  10. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  11. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  12. 2023RRSIS: Referring Remote Sensing Image Segmentationcited by 59
38 more
  1. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  2. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  3. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  4. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  5. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  6. 2024Toward Visual Grounding: A Surveycited by 52
  7. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
  8. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  9. 2023Mask Grounding for Referring Image Segmentationcited by 40
  10. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  11. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38
  12. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  13. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  14. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  15. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  16. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  17. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  18. 2024Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentationcited by 30
  19. 2023Universal Segmentation at Arbitrary Granularity with Language Instructioncited by 29
  20. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  21. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  22. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  23. 2024CoReS: Orchestrating the Dance of Reasoning and Segmentationcited by 25
  24. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Modelcited by 24
  25. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  26. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  27. 2025Multimodal Referring Segmentation: A Surveycited by 22
  28. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  29. 2024An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Groundingcited by 21
  30. 2023Extending CLIP's Image-Text Alignment to Referring Image Segmentationcited by 19
  31. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  32. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 18
  33. 2024SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentationcited by 17
  34. 2023Advancing Referring Expression Segmentation Beyond Single Imagecited by 16
  35. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructionscited by 16
  36. 2023RIO: A Benchmark for Reasoning Intention-Oriented Objects in Open Environmentscited by 15
  37. 2024PartGLEE: A Foundation Model for Recognizing and Parsing Any Objectscited by 15
  38. 2023EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignmentcited by 14

50 most cited of 147 All 147 (JSON)