# Clockwork Convnets for Video Semantic Segmentation

arXiv: https://arxiv.org/abs/1608.03609 · HTML: https://mlsanity.com/paper/1608.03609/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2016.

## Built on by (most cited first, 30 of 55)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [Semantic Segmentation using Vision Transformers: A survey](https://mlsanity.com/paper/2305.03273/) (2023)
- [Temporal Memory Attention for Video Semantic Segmentation](https://mlsanity.com/paper/2102.08643/) (2021)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [Skip-Convolutions for Efficient Video Processing](https://mlsanity.com/paper/2104.11487/) (2021)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [Mask Propagation for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2310.18954/) (2023)
- [Local Memory Attention for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2101.01715/) (2021)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos](https://mlsanity.com/paper/2203.03996/) (2022)
- [Clockwork Diffusion: Efficient Generation With Model-Step Distillation](https://mlsanity.com/paper/2312.08128/) (2023)
- [Exploiting Temporal State Space Sharing for Video Semantic Segmentation](https://mlsanity.com/paper/2503.20824/) (2025)
- [Improving Video Instance Segmentation via Temporal Pyramid Routing](https://mlsanity.com/paper/2107.13155/) (2021)
- [Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2401.15261/) (2024)
- [GSVNet: Guided Spatially-Varying Convolution for Fast Semantic Segmentation on Video](https://mlsanity.com/paper/2103.08834/) (2021)
- [Domain Adaptive Video Segmentation via Temporal Pseudo Supervision](https://mlsanity.com/paper/2207.02372/) (2022)
- [Unsupervised Domain Adaptation for Video Semantic Segmentation](https://mlsanity.com/paper/2107.11052/) (2021)
- [BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies](https://mlsanity.com/paper/2108.09376/) (2021)
- [Efficient Semantic Segmentation by Altering Resolutions for Compressed Videos](https://mlsanity.com/paper/2303.07224/) (2023)
- [Learning to Transfer for Traffic Forecasting via Multi-task Learning](https://mlsanity.com/paper/2111.15542/) (2021)
- [On Efficient Real-Time Semantic Segmentation: A Survey](https://mlsanity.com/paper/2206.08605/) (2022)
- [Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation](https://mlsanity.com/paper/2112.13815/) (2021)
- [Event Neural Networks](https://mlsanity.com/paper/2112.00891/) (2021)
- [MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos](https://mlsanity.com/paper/2210.09887/) (2022)

## Abstract (arXiv)

Recent years have seen tremendous progress in still-image segmentation; however the na"ive application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of "clockwork" convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.