mlsanityData to 7 Oct 2026

Predicting Deeper into the Future of Semantic Segmentation

arXiv 1703.07684 ↗First posted 22 Mar 2017cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

The ability to predict and therefore to anticipate the future is an important attribute of intelligence. It is also of utmost importance in real-time systems, e.g. in robotics or autonomous driving, which depend on visual scene understanding for decision making. While prediction of the raw RGB pixel values in future video frames has been studied in previous work, here we introduce the novel task of predicting semantic segmentations of future frames. Given a sequence of video frames, our goal is to predict segmentation maps of not yet observed video frames that lie up to a second or further in the future. We develop an autoregressive convolutional neural network that learns to iteratively generate multiple frames. Our results on the Cityscapes dataset show that directly predicting future segmentations is substantially better than predicting and then segmenting future RGB frames. Prediction results up to half a second in the future are visually convincing and are much more accurate than those of a baseline based on warping semantic segmentations using optical flow.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 21; 2022: 16; 2023: 3; 2024: 5; 2025: 3; 2026: 2.

51 papers.

Sort:

By area:

  1. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  2. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  3. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  4. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  5. 2021Predicting Video with VQVAEcited by 61
  6. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  7. 2019Imitation Learning for Human Pose Predictioncited by 44
  8. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
  9. 2026Learning Latent Action World Models In The Wildcited by 42
  10. 2022Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentationcited by 40
  11. 2021Semi-supervised Domain Adaptation based on Dual-level Domain Mixing for Semantic Segmentationcited by 33
  12. 2022Real-time Object Detection for Streaming Perceptioncited by 29
38 more
  1. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  2. 2022Optimizing Video Prediction via Video Frame Interpolationcited by 28
  3. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  4. 2021Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videoscited by 23
  5. 2021Learning Semantic-Aware Dynamics for Video Predictioncited by 21
  6. 2021Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentationcited by 20
  7. 2022Generative Adversarial Network for Future Hand Segmentation from Egocentric Videocited by 16
  8. 2021ECLIPSE : Envisioning CLoud Induced Perturbations in Solar Energycited by 14
  9. 2021Temporally smooth online action detection using cycle-consistent future anticipationcited by 13
  10. 2021Panoptic Segmentation Forecastingcited by 7
  11. 2023Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentationcited by 7
  12. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  13. 2025Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformerscited by 7
  14. 2022StreamYOLO: Real-time Object Detection for Streaming Perceptioncited by 6
  15. 2021Review of Video Predictive Understanding: Early Action Recognition and Future Action Predictioncited by 5
  16. 2021Recognizing Scenes from Novel Viewpointscited by 5
  17. 2022Forecasting Future Instance Segmentation with Learned Optical Flow and Warpingcited by 4
  18. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  19. 2021Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motioncited by 3
  20. 2021Hierarchical Video Generation for Complex Datacited by 3
  21. 2021Predicting Physical World Destinations for Commands Given to Self-Driving Carscited by 3
  22. 2022Joint Forecasting of Panoptic Segmentations with Difference Attentioncited by 3
  23. 2022Patch-based Object-centric Transformers for Efficient Video Generationcited by 3
  24. 2022Forecasting of depth and ego-motion with transformers and self-supervisioncited by 3
  25. 2022How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecastingcited by 3
  26. 2022DaDe: Delay-adaptive Detector for Streaming Perceptioncited by 3
  27. 2022Labeling Where Adapting Fails: Cross-Domain Semantic Segmentation with Point Supervision via Active Selectioncited by 2
  28. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  29. 2021Affine-modeled video extraction from a single motion blurred imagecited by 1
  30. 2021Semantic Prediction: Which One Should Come First, Recognition or Prediction?cited by 1
  31. 2023FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architecturescited by 1
  32. 2024Panoptic-Depth Forecastingcited by 1
  33. 2021Unsupervised Adaptive Semantic Segmentation with Local Lipschitz Constraintcited by 0
  34. 2021From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecastingcited by 0
  35. 2022Borrowing from yourself: Faster future video segmentation with partial channel updatecited by 0
  36. 2022Cascaded Video Generation for Videos In-the-Wildcited by 0
  37. 2023R2S100K: Road-Region Segmentation Dataset For Semi-Supervised Autonomous Driving in the Wildcited by 0
  38. 2024Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperationcited by 0

50 most cited of 51 All 51 (JSON)