# Region-based semantic segmentation with end-to-end training

arXiv: https://arxiv.org/abs/1607.07671 · HTML: https://mlsanity.com/paper/1607.07671/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2016.

## Built on by (most cited first, 9 of 9)

- [RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation](https://mlsanity.com/paper/2203.04187/) (2022)
- [ISDA: Position-Aware Instance Segmentation with Deformable Attention](https://mlsanity.com/paper/2202.12251/) (2022)
- [Automatic Volumetric Segmentation of Additive Manufacturing Defects with 3D U-Net](https://mlsanity.com/paper/2101.08993/) (2021)
- [OASIS: Automated Assessment of Urban Pedestrian Paths at Scale](https://mlsanity.com/paper/2303.02287/) (2023)
- [RMT-PPAD: Real-time Multi-task Learning for Panoptic Perception in Autonomous Driving](https://mlsanity.com/paper/2508.06529/) (2025)
- [A Novel Upsampling and Context Convolution for Image Semantic Segmentation](https://mlsanity.com/paper/2103.11110/) (2021)
- [Temporal-consistent CAMs for Weakly Supervised Video Segmentation in Waste Sorting](https://mlsanity.com/paper/2502.01455/) (2025)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [On the effectiveness of Rotation-Equivariance in U-Net: A Benchmark for Image Segmentation](https://mlsanity.com/paper/2412.09182/) (2024)

## Abstract (arXiv)

We propose a novel method for semantic segmentation, the task of labeling each pixel in an image with a semantic class. Our method combines the advantages of the two main competing paradigms. Methods based on region classification offer proper spatial support for appearance measurements, but typically operate in two separate stages, none of which targets pixel labeling performance at the end of the pipeline. More recent fully convolutional methods are capable of end-to-end training for the final pixel labeling, but resort to fixed patches as spatial support. We show how to modify modern region-based approaches to enable end-to-end training for semantic segmentation. This is achieved via a differentiable region-to-pixel layer and a differentiable free-form Region-of-Interest pooling layer. Our method improves the state-of-the-art in terms of class-average accuracy with 64.0% on SIFT Flow and 49.9% on PASCAL Context, and is particularly accurate at object boundaries.