# Detect, Replace, Refine: Deep Structured Prediction For Pixel Wise Labeling

arXiv: https://arxiv.org/abs/1612.04770 · HTML: https://mlsanity.com/paper/1612.04770/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2016.

## Built on by (most cited first, 11 of 11)

- [Open Challenges in Deep Stereo: the Booster Dataset](https://mlsanity.com/paper/2206.04671/) (2022)
- [Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces](https://mlsanity.com/paper/2301.08245/) (2023)
- [High Quality Segmentation for Ultra High-resolution Images](https://mlsanity.com/paper/2111.14482/) (2021)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Neural Disparity Refinement for Arbitrary Resolution Stereo](https://mlsanity.com/paper/2110.15367/) (2021)
- [A Unified Efficient Pyramid Transformer for Semantic Segmentation](https://mlsanity.com/paper/2107.14209/) (2021)
- [ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images](https://mlsanity.com/paper/2106.08107/) (2021)
- [RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation](https://mlsanity.com/paper/2206.07047/) (2022)
- [Sequential Ensembling for Semantic Segmentation](https://mlsanity.com/paper/2210.05387/) (2022)
- [PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction](https://mlsanity.com/paper/2111.15192/) (2021)
- [TransFiner: A Full-Scale Refinement Approach for Multiple Object Tracking](https://mlsanity.com/paper/2207.12967/) (2022)

## Abstract (arXiv)

Pixel wise image labeling is an interesting and challenging problem with great significance in the computer vision community. In order for a dense labeling algorithm to be able to achieve accurate and precise results, it has to consider the dependencies that exist in the joint space of both the input and the output variables. An implicit approach for modeling those dependencies is by training a deep neural network that, given as input an initial estimate of the output labels and the input image, it will be able to predict a new refined estimate for the labels. In this context, our work is concerned with what is the optimal architecture for performing the label improvement task. We argue that the prior approaches of either directly predicting new label estimates or predicting residual corrections w.r.t. the initial labels with feed-forward deep network architectures are sub-optimal. Instead, we propose a generic architecture that decomposes the label improvement task to three steps: 1) detecting the initial label estimates that are incorrect, 2) replacing the incorrect labels with new ones, and finally 3) refining the renewed labels by predicting residual corrections w.r.t. them. Furthermore, we explore and compare various other alternative architectures that consist of the aforementioned Detection, Replace, and Refine components. We extensively evaluate the examined architectures in the challenging task of dense disparity estimation (stereo matching) and we report both quantitative and qualitative results on three different datasets. Finally, our dense disparity estimation network that implements the proposed generic architecture, achieves state-of-the-art results in the KITTI 2015 test surpassing prior approaches by a significant margin.