# Convolutional Gated Recurrent Networks for Video Segmentation

arXiv: https://arxiv.org/abs/1611.05435 · HTML: https://mlsanity.com/paper/1611.05435/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2016.

## Built on by (most cited first, 27 of 27)

- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [RegNet: Self-Regulated Network for Image Classification](https://mlsanity.com/paper/2101.00590/) (2021)
- [Crop mapping from image time series: deep learning with multi-scale label hierarchies](https://mlsanity.com/paper/2102.08820/) (2021)
- [ZFlow: Gated Appearance Flow-based Virtual Try-on with 3D Priors](https://mlsanity.com/paper/2109.07001/) (2021)
- [Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview](https://mlsanity.com/paper/2105.14291/) (2021)
- [Event-based Simultaneous Localization and Mapping: A Comprehensive Survey](https://mlsanity.com/paper/2304.09793/) (2023)
- [ECLIPSE : Envisioning CLoud Induced Perturbations in Solar Energy](https://mlsanity.com/paper/2104.12419/) (2021)
- [High-Definition Map Generation Technologies For Autonomous Driving](https://mlsanity.com/paper/2206.05400/) (2022)
- [SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events](https://mlsanity.com/paper/2309.12842/) (2023)
- [Federated Variational Learning for Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2108.08404/) (2021)
- [Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting](https://mlsanity.com/paper/2101.02385/) (2021)
- [Learning to Estimate Two Dense Depths from LiDAR and Event Data](https://mlsanity.com/paper/2302.14444/) (2023)
- [Limb-Aware Virtual Try-On Network with Progressive Clothing Warping](https://mlsanity.com/paper/2503.14074/) (2025)
- [Automatic ultrasound vessel segmentation with deep spatiotemporal context learning](https://mlsanity.com/paper/2111.02461/) (2021)
- [PCT-CycleGAN: Paired Complementary Temporal Cycle-Consistent Adversarial Networks for Radar-Based Precipitation Nowcasting](https://mlsanity.com/paper/2211.15046/) (2022)
- [Progressive Limb-Aware Virtual Try-On](https://mlsanity.com/paper/2503.12588/) (2025)
- [Learning to Track Object Position through Occlusion](https://mlsanity.com/paper/2106.10766/) (2021)
- [Uncovering wall-shear stress dynamics from neural-network enhanced fluid flow measurements](https://mlsanity.com/paper/2310.11147/) (2023)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion](https://mlsanity.com/paper/2405.19921/) (2024)
- [A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentation](https://mlsanity.com/paper/2505.19159/) (2025)
- [Finding a Needle in a Haystack: Tiny Flying Object Detection in 4K Videos using a Joint Detection-and-Tracking Approach](https://mlsanity.com/paper/2105.08253/) (2021)
- [SGE net: Video object detection with squeezed GRU and information entropy map](https://mlsanity.com/paper/2106.07224/) (2021)
- [Recurrently Estimating Reflective Symmetry Planes from Partial Pointclouds](https://mlsanity.com/paper/2106.16129/) (2021)
- [Cycle-Consistent World Models for Domain Independent Latent Imagination](https://mlsanity.com/paper/2110.00808/) (2021)
- [Feature-aggregated spatiotemporal spine surface estimation for wearable patch ultrasound volumetric imaging](https://mlsanity.com/paper/2211.05962/) (2022)
- [Temporal-Aware Refinement for Video-based Human Pose and Shape Recovery](https://mlsanity.com/paper/2311.09543/) (2023)

## Abstract (arXiv)

Semantic segmentation has recently witnessed major progress, where fully convolutional neural networks have shown to perform well. However, most of the previous work focused on improving single image segmentation. To our knowledge, no prior work has made use of temporal video information in a recurrent network. In this paper, we introduce a novel approach to implicitly utilize temporal data in videos for online semantic segmentation. The method relies on a fully convolutional network that is embedded into a gated recurrent architecture. This design receives a sequence of consecutive video frames and outputs the segmentation of the last frame. Convolutional gated recurrent networks are used for the recurrent part to preserve spatial connectivities in the image. Our proposed method can be applied in both online and batch segmentation. This architecture is tested for both binary and semantic video segmentation tasks. Experiments are conducted on the recent benchmarks in SegTrack V2, Davis, CityScapes, and Synthia. Using recurrent fully convolutional networks improved the baseline network performance in all of our experiments. Namely, 5% and 3% improvement of F-measure in SegTrack2 and Davis respectively, 5.7% improvement in mean IoU in Synthia and 3.5% improvement in categorical mean IoU in CityScapes. The performance of the RFCN network depends on its baseline fully convolutional network. Thus RFCN architecture can be seen as a method to improve its baseline segmentation network by exploiting spatiotemporal information in videos.