Bridging Object Detection and Segmentation with Polygon Detection Transformers

Box detection and mask segmentation are two dominant paradigms for foreground representation: boxes are efficient but too coarse for object shapes, while masks are accurate but over-modeled for compact geometry. To bridge this gap, we present a Polygon Detection Transformer (Poly-DETR) built upon Polar Representation, where object queries regress a starting point and its fixed number of radial distances to directly construct the contour-approximating polygon. This formulation can be integrated into most DETR-like detectors by linear extension, since box is a degenerate case of Polar Representation with four rays. Furthermore, we propose two simple but necessary designs, Polar Deformable Attention and Position-Aware Training Scheme, to align feature sampling and polygon supervision. As a DETR-oriented advancement of Polar Representation, Poly-DETR outperforms existing polar-based methods by 4.7 mAP on MS COCO. Moreover, we explore the application regimes of polygon detection in geometry-driven domains, including remote sensing, medical imaging, and autonomous driving. In particular, Poly-DETR shows stronger scalability than its mask-based counterpart in high-resolution scenarios. Additional experiments show that, owing to its Transformer structure, Poly-DETR can be naturally extended to recent DETR variants equipped with foundation-model priors.