# Learning to Refine Object Segments

arXiv: https://arxiv.org/abs/1603.08695 · HTML: https://mlsanity.com/paper/1603.08695/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2016.

## Built on by (most cited first, 30 of 67)

- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Class-agnostic Object Detection with Multi-modal Transformer](https://mlsanity.com/paper/2111.11430/) (2021)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features](https://mlsanity.com/paper/2104.08569/) (2021)
- [Teaching Matters: Investigating the Role of Supervision in Vision Transformers](https://mlsanity.com/paper/2212.03862/) (2022)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [EDTER: Edge Detection with Transformer](https://mlsanity.com/paper/2203.08566/) (2022)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition](https://mlsanity.com/paper/2104.05367/) (2021)
- [BoxeR: Box-Attention for 2D and 3D Transformers](https://mlsanity.com/paper/2111.13087/) (2021)
- [Instance Shadow Detection with A Single-Stage Detector](https://mlsanity.com/paper/2207.04614/) (2022)
- [D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2111.07774/) (2021)
- [ContrastMask: Contrastive Learning to Segment Every Thing](https://mlsanity.com/paper/2203.09775/) (2022)
- [Image Amodal Completion: A Survey](https://mlsanity.com/paper/2207.02062/) (2022)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [A^2-FPN: Attention Aggregation based Feature Pyramid Network for Instance Segmentation](https://mlsanity.com/paper/2105.03186/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review](https://mlsanity.com/paper/2203.10939/) (2022)
- [SiamMask: A Framework for Fast Online Object Tracking and Segmentation](https://mlsanity.com/paper/2207.02088/) (2022)
- [AI-based Aortic Vessel Tree Segmentation for Cardiovascular Diseases Treatment: Status Quo](https://mlsanity.com/paper/2108.02998/) (2021)
- [One-Trimap Video Matting](https://mlsanity.com/paper/2207.13353/) (2022)
- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [Attention mechanisms and deep learning for machine vision: A survey of the state of the art](https://mlsanity.com/paper/2106.07550/) (2021)
- [A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture](https://mlsanity.com/paper/2210.11318/) (2022)
- [Online Unsupervised Video Object Segmentation via Contrastive Motion Clustering](https://mlsanity.com/paper/2306.12048/) (2023)
- [Localizing Small Apples in Complex Apple Orchard Environments](https://mlsanity.com/paper/2202.11372/) (2022)

## Abstract (arXiv)

Object segmentation requires both object-level information and low-level pixel data. This presents a challenge for feedforward networks: lower layers in convolutional nets capture rich spatial information, while upper layers encode object-level knowledge but are invariant to factors such as pose and appearance. In this work we propose to augment feedforward nets for object segmentation with a novel top-down refinement approach. The resulting bottom-up/top-down architecture is capable of efficiently generating high-fidelity object masks. Similarly to skip connections, our approach leverages features at all layers of the net. Unlike skip connections, our approach does not attempt to output independent predictions at each layer. Instead, we first output a coarse `mask encoding' in a feedforward pass, then refine this mask encoding in a top-down pass utilizing features at successively lower layers. The approach is simple, fast, and effective. Building on the recent DeepMask network for generating object proposals, we show accuracy improvements of 10-20% in average recall for various setups. Additionally, by optimizing the overall network architecture, our approach, which we call SharpMask, is 50% faster than the original DeepMask network (under .8s per image).