# Recurrent Scene Parsing with Perspective Understanding in the Loop

arXiv: https://arxiv.org/abs/1705.07238 · HTML: https://mlsanity.com/paper/1705.07238/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2017.

## Built on by (most cited first, 16 of 16)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes](https://mlsanity.com/paper/2101.06085/) (2021)
- [Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2104.13613/) (2021)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [Polarized Self-Attention: Towards High-quality Pixel-wise Regression](https://mlsanity.com/paper/2107.00782/) (2021)
- [Adaptivity and Modularity for Efficient Generalization Over Task Complexity](https://mlsanity.com/paper/2310.08866/) (2023)
- [Memory-based Semantic Segmentation for Off-road Unstructured Natural Environments](https://mlsanity.com/paper/2108.05635/) (2021)
- [Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation](https://mlsanity.com/paper/2201.01427/) (2022)
- [Deep feature selection-and-fusion for RGB-D semantic segmentation](https://mlsanity.com/paper/2105.04102/) (2021)
- [U-HRNet: Delving into Improving Semantic Representation of High Resolution Network for Dense Prediction](https://mlsanity.com/paper/2210.07140/) (2022)
- [A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design](https://mlsanity.com/paper/2303.04315/) (2023)
- [Hierarchical Pyramid Representations for Semantic Segmentation](https://mlsanity.com/paper/2104.01792/) (2021)
- [Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task](https://mlsanity.com/paper/2105.13531/) (2021)
- [TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency](https://mlsanity.com/paper/2210.05152/) (2022)
- [OVeNet: Offset Vector Network for Semantic Segmentation](https://mlsanity.com/paper/2303.14516/) (2023)
- [Scale Invariant Semantic Segmentation with RGB-D Fusion](https://mlsanity.com/paper/2204.04679/) (2022)

## Abstract (arXiv)

Objects may appear at arbitrary scales in perspective images of a scene, posing a challenge for recognition systems that process images at a fixed resolution. We propose a depth-aware gating module that adaptively selects the pooling field size in a convolutional network architecture according to the object scale (inversely proportional to the depth) so that small details are preserved for distant objects while larger receptive fields are used for those nearby. The depth gating signal is provided by stereo disparity or estimated directly from monocular input. We integrate this depth-aware gating into a recurrent convolutional neural network to perform semantic segmentation. Our recurrent module iteratively refines the segmentation results, leveraging the depth and semantic predictions from the previous iterations. Through extensive experiments on four popular large-scale RGB-D datasets, we demonstrate this approach achieves competitive semantic segmentation performance with a model which is substantially more compact. We carry out extensive analysis of this architecture including variants that operate on monocular RGB but use depth as side-information during training, unsupervised gating as a generic attentional mechanism, and multi-resolution gating. We find that gated pooling for joint semantic segmentation and depth yields state-of-the-art results for quantitative monocular depth estimation.