# Panoptic Image Annotation with a Collaborative Assistant

arXiv: https://arxiv.org/abs/1906.06798 · HTML: https://mlsanity.com/paper/1906.06798/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2019.

## Built on by (most cited first, 3 of 3)

- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Evaluating and Improving Interactions with Hazy Oracles](https://mlsanity.com/paper/2110.10206/) (2021)
- [It's Not Just Labeling -- A Research on LLM Generated Feedback Interpretability and Image Labeling Sketch Features](https://mlsanity.com/paper/2505.19419/) (2025)

## Abstract (arXiv)

This paper aims to reduce the time to annotate images for panoptic segmentation, which requires annotating segmentation masks and class labels for all object instances and stuff regions. We formulate our approach as a collaborative process between an annotator and an automated assistant who take turns to jointly annotate an image using a predefined pool of segments. Actions performed by the annotator serve as a strong contextual signal. The assistant intelligently reacts to this signal by annotating other parts of the image on its own, which reduces the amount of work required by the annotator. We perform thorough experiments on the COCO panoptic dataset, both in simulation and with human annotators. These demonstrate that our approach is significantly faster than the recent machine-assisted interface of [4], and 2.4x to 5x faster than manual polygon drawing. Finally, we show on ADE20k that our method can be used to efficiently annotate new datasets, bootstrapping from a very small amount of annotated data.