# Segmenting the Future

arXiv: https://arxiv.org/abs/1904.10666 · HTML: https://mlsanity.com/paper/1904.10666/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 19 of 19)

- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Back to the Features: DINO as a Foundation for Video World Models](https://mlsanity.com/paper/2507.19468/) (2025)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Real-time Object Detection for Streaming Perception](https://mlsanity.com/paper/2203.12338/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions](https://mlsanity.com/paper/2302.11850/) (2023)
- [Generative Adversarial Network for Future Hand Segmentation from Egocentric Video](https://mlsanity.com/paper/2203.11305/) (2022)
- [Long-term Video Frame Interpolation via Feature Propagation](https://mlsanity.com/paper/2203.15427/) (2022)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [StreamYOLO: Real-time Object Detection for Streaming Perception](https://mlsanity.com/paper/2207.10433/) (2022)
- [Instance-Aware Predictive Navigation in Multi-Agent Environments](https://mlsanity.com/paper/2101.05893/) (2021)
- [Forecasting Future Instance Segmentation with Learned Optical Flow and Warping](https://mlsanity.com/paper/2211.08049/) (2022)
- [Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion](https://mlsanity.com/paper/2101.10777/) (2021)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [Forecasting of depth and ego-motion with transformers and self-supervision](https://mlsanity.com/paper/2206.07435/) (2022)
- [DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle](https://mlsanity.com/paper/2307.06647/) (2023)
- [TextOCVP: Object-Centric Video Prediction with Language Guidance](https://mlsanity.com/paper/2502.11655/) (2025)
- [From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting](https://mlsanity.com/paper/2107.10068/) (2021)
- [Borrowing from yourself: Faster future video segmentation with partial channel update](https://mlsanity.com/paper/2202.05748/) (2022)

## Abstract (arXiv)

Predicting the future is an important aspect for decision-making in robotics or autonomous driving systems, which heavily rely upon visual scene understanding. While prior work attempts to predict future video pixels, anticipate activities or forecast future scene semantic segments from segmentation of the preceding frames, methods that predict future semantic segmentation solely from the previous frame RGB data in a single end-to-end trainable model do not exist. In this paper, we propose a temporal encoder-decoder network architecture that encodes RGB frames from the past and decodes the future semantic segmentation. The network is coupled with a new knowledge distillation training framework specific for the forecasting task. Our method, only seeing preceding video frames, implicitly models the scene segments while simultaneously accounting for the object dynamics to infer the future scene semantic segments. Our results on Cityscapes and Apolloscape outperform the baseline and current state-of-the-art methods. Code is available at https://github.com/eddyhkchiu/segmenting_the_future/.