mlsanityData to 14 Oct 2026

Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion

arXiv 2101.10777 ↗First posted 26 Jan 2021cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Dense semantic forecasting anticipates future events in video by inferring pixel-level semantics of an unobserved future image. We present a novel approach that is applicable to various single-frame architectures and tasks. Our approach consists of two modules. Feature-to-motion (F2M) module forecasts a dense deformation field that warps past features into their future positions. Feature-to-feature (F2F) module regresses the future features directly and is therefore able to account for emergent scenery. The compound F2MF model decouples the effects of motion from the effects of novelty in a task-agnostic manner. We aim to apply F2MF forecasting to the most subsampled and the most abstract representation of a desired single-frame model. Our design takes advantage of deformable convolutions and spatial correlation coefficients across neighbouring time instants. We perform experiments on three dense prediction tasks: semantic segmentation, instance-level segmentation, and panoptic segmentation. The results reveal state-of-the-art forecasting accuracy across three dense prediction tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

34 of 56 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Mask R-CNNcited by 7,277
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  7. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  8. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 670
  9. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  10. 2019Panoptic-DeepLabcited by 363
  11. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  12. 2015Stereo Matching by Training a Convolutional Neural Network to Compare Image Patchescited by 151
22 more
  1. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  2. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  3. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  4. 2019In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Imagescited by 100
  5. 2020Learning Fast and Robust Target Models for Video Object Segmentationcited by 76
  6. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 55
  7. 2019Video Generation from Single Semantic Label Mapcited by 54
  8. 2017Predicting Deeper into the Future of Semantic Segmentationcited by 51
  9. 2018SDCNet: Video Prediction Using Spatially-Displaced Convolutioncited by 50
  10. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  11. 2018Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systemscited by 42
  12. 2018Predicting Future Instance Segmentation by Forecasting Convolutional Featurescited by 37
  13. 2017Predicting Scene Parsing and Motion Dynamics in the Futurecited by 20
  14. 2018Recurrent Flow-Guided Semantic Forecastingcited by 19
  15. 2019Segmenting the Futurecited by 19
  16. 2018Bayesian Prediction of Future Street Scenes using Synthetic Likelihoodscited by 17
  17. 2019Efficient Ladder-style DenseNets for Semantic Segmentation of Large Imagescited by 17
  18. 2015Anticipating Visual Representations from Unlabeled Videocited by 14
  19. 2018Future Semantic Segmentation with Convolutional LSTMcited by 14
  20. 2019Single Level Feature-to-Feature Forecasting with Deformable Convolutionscited by 10
  21. 2018Future Segmentation Using 3D Structurecited by 5
  22. 2017One-Step Time-Dependent Future Video Frame Prediction with a Convolutional Encoder-Decoder Neural Networkcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 1.

3 papers.

  1. 2025Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformerscited by 8
  2. 2022Joint Forecasting of Panoptic Segmentations with Difference Attentioncited by 3
  3. 2024Panoptic-Depth Forecastingcited by 1