# Annotating Object Instances with a Polygon-RNN

arXiv: https://arxiv.org/abs/1704.05548 · HTML: https://mlsanity.com/paper/1704.05548/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2017.

## Built on by (most cited first, 30 of 88)

- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [VectorMapNet: End-to-end Vectorized HD Map Learning](https://mlsanity.com/paper/2206.08920/) (2022)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [STEP: Segmenting and Tracking Every Pixel](https://mlsanity.com/paper/2102.11859/) (2021)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning](https://mlsanity.com/paper/2104.12166/) (2021)
- [PolyWorld: Polygonal Building Extraction with Graph Neural Networks in Satellite Images](https://mlsanity.com/paper/2111.15491/) (2021)
- [Boundary-Aware Segmentation Network for Mobile and Web Applications](https://mlsanity.com/paper/2101.04704/) (2021)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [All you need are a few pixels: semantic segmentation with PixelPick](https://mlsanity.com/paper/2104.06394/) (2021)
- [TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning](https://mlsanity.com/paper/2406.15658/) (2024)
- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [A Spacecraft Dataset for Detection, Segmentation and Parts Recognition](https://mlsanity.com/paper/2106.08186/) (2021)
- [Towards General-Purpose Representation Learning of Polygonal Geometries](https://mlsanity.com/paper/2209.15458/) (2022)
- [iCurb: Imitation Learning-based Detection of Road Curbs using Aerial Images for Autonomous Driving](https://mlsanity.com/paper/2103.17118/) (2021)
- [PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection](https://mlsanity.com/paper/2212.04708/) (2022)
- [Learning the What and How of Annotation in Video Object Segmentation](https://mlsanity.com/paper/2311.04414/) (2023)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [ScalableMap: Scalable Map Learning for Online Long-Range Vectorized HD Map Construction](https://mlsanity.com/paper/2310.13378/) (2023)
- [Structure Embedded Nucleus Classification for Histopathology Images](https://mlsanity.com/paper/2302.11416/) (2023)
- [Interactive Segmentation as Gaussian Process Classification](https://mlsanity.com/paper/2302.14578/) (2023)
- [DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer](https://mlsanity.com/paper/2304.06668/) (2023)

## Abstract (arXiv)

We propose an approach for semi-automatic annotation of object instances. While most current methods treat object segmentation as a pixel-labeling problem, we here cast it as a polygon prediction task, mimicking how most current datasets have been annotated. In particular, our approach takes as input an image crop and sequentially produces vertices of the polygon outlining the object. This allows a human annotator to interfere at any time and correct a vertex if needed, producing as accurate segmentation as desired by the annotator. We show that our approach speeds up the annotation process by a factor of 4.7 across all classes in Cityscapes, while achieving 78.4% agreement in IoU with original ground-truth, matching the typical agreement between human annotators. For cars, our speed-up factor is 7.3 for an agreement of 82.2%. We further show generalization capabilities of our approach to unseen datasets.