# Efficient Semantic Video Segmentation with Per-frame Inference

arXiv: https://arxiv.org/abs/2002.11433 · HTML: https://mlsanity.com/paper/2002.11433/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 30 of 63)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [Temporal Memory Attention for Video Semantic Segmentation](https://mlsanity.com/paper/2102.08643/) (2021)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Stubborn: A Strong Baseline for Indoor Object Navigation](https://mlsanity.com/paper/2203.07359/) (2022)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [Mask Propagation for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2310.18954/) (2023)
- [Local Memory Attention for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2101.01715/) (2021)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency](https://mlsanity.com/paper/2103.12886/) (2021)
- [MarS3D: A Plug-and-Play Motion-Aware Model for Semantic Segmentation on Multi-Scan 3D Point Clouds](https://mlsanity.com/paper/2307.09316/) (2023)
- [Boosting Object Representation Learning via Motion and Object Continuity](https://mlsanity.com/paper/2211.09771/) (2022)
- [A spatio-temporal network for video semantic segmentation in surgical videos](https://mlsanity.com/paper/2306.11052/) (2023)
- [Exploiting Temporal State Space Sharing for Video Semantic Segmentation](https://mlsanity.com/paper/2503.20824/) (2025)
- [Biomedical SAM 2: Segment Anything in Biomedical Images and Videos](https://mlsanity.com/paper/2408.03286/) (2024)
- [Perceptual Consistency in Video Segmentation](https://mlsanity.com/paper/2110.12385/) (2021)
- [Video Shadow Detection via Spatio-Temporal Interpolation Consistency Training](https://mlsanity.com/paper/2206.08801/) (2022)
- [Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2401.15261/) (2024)
- [Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models](https://mlsanity.com/paper/2405.16947/) (2024)
- [RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation](https://mlsanity.com/paper/2203.04187/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Pseudo Supervision](https://mlsanity.com/paper/2207.02372/) (2022)
- [Temporal Context for Robust Maritime Obstacle Detection](https://mlsanity.com/paper/2203.05352/) (2022)
- [Delta Distillation for Efficient Video Processing](https://mlsanity.com/paper/2203.09594/) (2022)
- [MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2209.04471/) (2022)
- [Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free Domain Adaptation for Video Semantic Segmentation](https://mlsanity.com/paper/2303.14361/) (2023)
- [Unsupervised Domain Adaptation for Video Semantic Segmentation](https://mlsanity.com/paper/2107.11052/) (2021)
- [BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies](https://mlsanity.com/paper/2108.09376/) (2021)

## Abstract (arXiv)

For semantic segmentation, most existing real-time deep models trained with each frame independently may produce inconsistent results for a video sequence. Advanced methods take into considerations the correlations in the video sequence, e.g., by propagating the results to the neighboring frames using optical flow, or extracting the frame representations with other frames, which may lead to inaccurate results or unbalanced latency. In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. Different from previous per-frame models, we explicitly consider the temporal consistency among frames as extra constraints during the training process and embed the temporal consistency into the segmentation network. Therefore, in the inference process, we can process each frame independently with no latency, and improve the temporal consistency with no extra computational cost and post-processing. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed. Our results outperform previous keyframe based methods with a better trade-off between the accuracy and the inference speed on popular benchmarks, including the Cityscapes and Camvid. The temporal consistency is also improved compared with corresponding baselines which are trained with each frame independently. Code is available at: https://tinyurl.com/segment-video