# Forecasting Future Instance Segmentation with Learned Optical Flow and Warping

arXiv: https://arxiv.org/abs/2211.08049 · HTML: https://mlsanity.com/paper/2211.08049/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Predicting Deeper into the Future of Semantic Segmentation](https://mlsanity.com/paper/1703.07684/) (2017)
- [Predicting Future Instance Segmentation by Forecasting Convolutional Features](https://mlsanity.com/paper/1803.11496/) (2018)
- [BirdNet: a 3D Object Detection Framework from LiDAR information](https://mlsanity.com/paper/1805.01195/) (2018)
- [Illuminating Pedestrians via Simultaneous Detection & Segmentation](https://mlsanity.com/paper/1706.08564/) (2017)
- [Recurrent Flow-Guided Semantic Forecasting](https://mlsanity.com/paper/1809.08318/) (2018)
- [Segmenting the Future](https://mlsanity.com/paper/1904.10666/) (2019)
- [Single Level Feature-to-Feature Forecasting with Deformable Convolutions](https://mlsanity.com/paper/1907.11475/) (2019)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [Segmentation Free Object Discovery in Video](https://mlsanity.com/paper/1609.00221/) (2016)
- [Semantic Road Layout Understanding by Generative Adversarial Inpainting](https://mlsanity.com/paper/1805.11746/) (2018)

## Built on by (most cited first, 4 of 4)

- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers](https://mlsanity.com/paper/2501.08303/) (2025)
- [FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures](https://mlsanity.com/paper/2310.20593/) (2023)
- [Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time](https://mlsanity.com/paper/2212.08058/) (2022)

## Abstract (arXiv)

For an autonomous vehicle it is essential to observe the ongoing dynamics of a scene and consequently predict imminent future scenarios to ensure safety to itself and others. This can be done using different sensors and modalities. In this paper we investigate the usage of optical flow for predicting future semantic segmentations. To do so we propose a model that forecasts flow fields autoregressively. Such predictions are then used to guide the inference of a learned warping function that moves instance segmentations on to future frames. Results on the Cityscapes dataset demonstrate the effectiveness of optical-flow methods.