# Recurrent Flow-Guided Semantic Forecasting

arXiv: https://arxiv.org/abs/1809.08318 · HTML: https://mlsanity.com/paper/1809.08318/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2018.

## Built on by (most cited first, 19 of 19)

- [Diffusion Models for Video Prediction and Infilling](https://mlsanity.com/paper/2206.07696/) (2022)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers](https://mlsanity.com/paper/2501.08303/) (2025)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [Forecasting Future Instance Segmentation with Learned Optical Flow and Warping](https://mlsanity.com/paper/2211.08049/) (2022)
- [Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion](https://mlsanity.com/paper/2101.10777/) (2021)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [Forecasting of depth and ego-motion with transformers and self-supervision](https://mlsanity.com/paper/2206.07435/) (2022)
- [How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecasting](https://mlsanity.com/paper/2207.00506/) (2022)
- [FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures](https://mlsanity.com/paper/2310.20593/) (2023)
- [Flow-guided Motion Prediction with Semantics and Dynamic Occupancy Grid Maps](https://mlsanity.com/paper/2407.15675/) (2024)
- [Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling](https://mlsanity.com/paper/2504.05537/) (2025)
- [Enhancing Bandwidth Efficiency for Video Motion Transfer Applications using Deep Learning Based Keypoint Prediction](https://mlsanity.com/paper/2403.11337/) (2024)
- [Taylor saves for later: disentanglement for video prediction using Taylor representation](https://mlsanity.com/paper/2105.11062/) (2021)
- [From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting](https://mlsanity.com/paper/2107.10068/) (2021)
- [Borrowing from yourself: Faster future video segmentation with partial channel update](https://mlsanity.com/paper/2202.05748/) (2022)
- [Enabling Real-time Neural Recovery for Cloud Gaming on Mobile Devices](https://mlsanity.com/paper/2307.07847/) (2023)
- [Real-Time Neural Video Recovery and Enhancement on Mobile Devices](https://mlsanity.com/paper/2307.12152/) (2023)

## Abstract (arXiv)

Understanding the world around us and making decisions about the future is a critical component to human intelligence. As autonomous systems continue to develop, their ability to reason about the future will be the key to their success. Semantic anticipation is a relatively under-explored area for which autonomous vehicles could take advantage of (e.g., forecasting pedestrian trajectories). Motivated by the need for real-time prediction in autonomous systems, we propose to decompose the challenging semantic forecasting task into two subtasks: current frame segmentation and future optical flow prediction. Through this decomposition, we built an efficient, effective, low overhead model with three main components: flow prediction network, feature-flow aggregation LSTM, and end-to-end learnable warp layer. Our proposed method achieves state-of-the-art accuracy on short-term and moving objects semantic forecasting while simultaneously reducing model parameters by up to 95% and increasing efficiency by greater than 40x.