# Video Scene Parsing with Predictive Feature Learning

arXiv: https://arxiv.org/abs/1612.00119 · HTML: https://mlsanity.com/paper/1612.00119/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2016.

## Built on by (most cited first, 23 of 23)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Polarized Self-Attention: Towards High-quality Pixel-wise Regression](https://mlsanity.com/paper/2107.00782/) (2021)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation](https://mlsanity.com/paper/2203.04187/) (2022)
- [Cross-attention Spatio-temporal Context Transformer for Semantic Segmentation of Historical Maps](https://mlsanity.com/paper/2310.12616/) (2023)
- [Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation](https://mlsanity.com/paper/2309.08020/) (2023)
- [Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation](https://mlsanity.com/paper/2112.13815/) (2021)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Deep Common Feature Mining for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2403.02689/) (2024)
- [Towards Open-Vocabulary Video Semantic Segmentation](https://mlsanity.com/paper/2412.09329/) (2024)
- [How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecasting](https://mlsanity.com/paper/2207.00506/) (2022)
- [U-HRNet: Delving into Improving Semantic Representation of High Resolution Network for Dense Prediction](https://mlsanity.com/paper/2210.07140/) (2022)
- [Motion-state Alignment for Video Semantic Segmentation](https://mlsanity.com/paper/2304.08820/) (2023)
- [Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation](https://mlsanity.com/paper/2604.10950/) (2026)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [OVeNet: Offset Vector Network for Semantic Segmentation](https://mlsanity.com/paper/2303.14516/) (2023)
- [Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction](https://mlsanity.com/paper/2604.11707/) (2026)
- [From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting](https://mlsanity.com/paper/2107.10068/) (2021)
- [Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation](https://mlsanity.com/paper/2206.09604/) (2022)
- [SA-VIS: Sparse frame Annotations for training Video Instance Segmentation](https://mlsanity.com/paper/2606.20140/) (2026)

## Abstract (arXiv)

In this work, we address the challenging video scene parsing problem by developing effective representation learning methods given limited parsing annotations. In particular, we contribute two novel methods that constitute a unified parsing framework. (1) Predictive feature learning from nearly unlimited unlabeled video data. Different from existing methods learning features from single frame parsing, we learn spatiotemporal discriminative features by enforcing a parsing network to predict future frames and their parsing maps (if available) given only historical frames. In this way, the network can effectively learn to capture video dynamics and temporal context, which are critical clues for video scene parsing, without requiring extra manual annotations. (2) Prediction steering parsing architecture that effectively adapts the learned spatiotemporal features to scene parsing tasks and provides strong guidance for any off-the-shelf parsing model to achieve better video scene parsing performance. Extensive experiments over two challenging datasets, Cityscapes and Camvid, have demonstrated the effectiveness of our methods by showing significant improvement over well-established baselines.