# Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation

arXiv: https://arxiv.org/abs/2104.05239 · HTML: https://mlsanity.com/paper/2104.05239/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 56 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)

## Built on by (most cited first, 30 of 43)

- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Bilateral Reference for High-Resolution Dichotomous Image Segmentation](https://mlsanity.com/paper/2401.03407/) (2024)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Mask Transfiner for High-Quality Instance Segmentation](https://mlsanity.com/paper/2111.13673/) (2021)
- [Sparse Instance Activation for Real-Time Instance Segmentation](https://mlsanity.com/paper/2203.12827/) (2022)
- [MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2404.00876/) (2024)
- [FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation](https://mlsanity.com/paper/2303.08594/) (2023)
- [SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process](https://mlsanity.com/paper/2312.12425/) (2023)
- [Video Mask Transfiner for High-Quality Video Instance Segmentation](https://mlsanity.com/paper/2207.14012/) (2022)
- [FADE: Fusing the Assets of Decoder and Encoder for Task-Agnostic Upsampling](https://mlsanity.com/paper/2207.10392/) (2022)
- [SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement](https://mlsanity.com/paper/2502.06756/) (2025)
- [Active Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2207.11493/) (2022)
- [SharpContour: A Contour-based Boundary Refinement Approach for Efficient and Accurate Instance Segmentation](https://mlsanity.com/paper/2203.13312/) (2022)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Visual Recognition by Request](https://mlsanity.com/paper/2207.14227/) (2022)
- [Recurrent Generic Contour-based Instance Segmentation with Progressive Learning](https://mlsanity.com/paper/2301.08898/) (2023)
- [Milestones in Autonomous Driving and Intelligent Vehicles Part II: Perception and Planning](https://mlsanity.com/paper/2306.01980/) (2023)
- [PatchDCT: Patch Refinement for High Quality Instance Segmentation](https://mlsanity.com/paper/2302.02693/) (2023)
- [Training-Free Dataset Pruning for Instance Segmentation](https://mlsanity.com/paper/2503.00828/) (2025)
- [Class-Aware Cartilage Segmentation for Autonomous US-CT Registration in Robotic Intercostal Ultrasound Imaging](https://mlsanity.com/paper/2406.04100/) (2024)
- [Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations](https://mlsanity.com/paper/2406.10114/) (2024)
- [Uncertainty-Masked Bernoulli Diffusion for Camouflaged Object Detection Refinement](https://mlsanity.com/paper/2506.10712/) (2025)
- [Self-supervised Learning for Panoptic Segmentation of Multiple Fruit Flower Species](https://mlsanity.com/paper/2209.04618/) (2022)
- [FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures](https://mlsanity.com/paper/2407.13500/) (2024)
- [Cell Instance Segmentation: The Devil Is in the Boundaries](https://mlsanity.com/paper/2510.09848/) (2025)
- [PVPUFormer: Probabilistic Visual Prompt Unified Transformer for Interactive Image Segmentation](https://mlsanity.com/paper/2306.06656/) (2023)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [GeloVec: Higher Dimensional Geometric Smoothing for Coherent Visual Feature Extraction in Image Segmentation](https://mlsanity.com/paper/2505.01057/) (2025)
- [rNCA: Self-Repairing Segmentation Masks](https://mlsanity.com/paper/2512.13397/) (2025)

## Abstract (arXiv)

Tremendous efforts have been made on instance segmentation but the mask quality is still not satisfactory. The boundaries of predicted instance masks are usually imprecise due to the low spatial resolution of feature maps and the imbalance problem caused by the extremely low proportion of boundary pixels. To address these issues, we propose a conceptually simple yet effective post-processing refinement framework to improve the boundary quality based on the results of any instance segmentation model, termed BPR. Following the idea of looking closer to segment boundaries better, we extract and refine a series of small boundary patches along the predicted instance boundaries. The refinement is accomplished by a boundary patch refinement network at higher resolution. The proposed BPR framework yields significant improvements over the Mask R-CNN baseline on Cityscapes benchmark, especially on the boundary-aware metrics. Moreover, by applying the BPR framework to the PolyTransform + SegFix baseline, we reached 1st place on the Cityscapes leaderboard.