# Future Semantic Segmentation with Convolutional LSTM

arXiv: https://arxiv.org/abs/1807.07946 · HTML: https://mlsanity.com/paper/1807.07946/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2018.

## Built on by (most cited first, 14 of 14)

- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos](https://mlsanity.com/paper/2107.09504/) (2021)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Generative Adversarial Network for Future Hand Segmentation from Egocentric Video](https://mlsanity.com/paper/2203.11305/) (2022)
- [Exploiting Temporal State Space Sharing for Video Semantic Segmentation](https://mlsanity.com/paper/2503.20824/) (2025)
- [Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers](https://mlsanity.com/paper/2501.08303/) (2025)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [Instance-Aware Predictive Navigation in Multi-Agent Environments](https://mlsanity.com/paper/2101.05893/) (2021)
- [Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion](https://mlsanity.com/paper/2101.10777/) (2021)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction](https://mlsanity.com/paper/2604.11707/) (2026)
- [Borrowing from yourself: Faster future video segmentation with partial channel update](https://mlsanity.com/paper/2202.05748/) (2022)
- [Shift-Memory Network for Temporal Scene Segmentation](https://mlsanity.com/paper/2202.08399/) (2022)
- [Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models](https://mlsanity.com/paper/2610.01942/) (2026)

## Abstract (arXiv)

We consider the problem of predicting semantic segmentation of future frames in a video. Given several observed frames in a video, our goal is to predict the semantic segmentation map of future frames that are not yet observed. A reliable solution to this problem is useful in many applications that require real-time decision making, such as autonomous driving. We propose a novel model that uses convolutional LSTM (ConvLSTM) to encode the spatiotemporal information of observed frames for future prediction. We also extend our model to use bidirectional ConvLSTM to capture temporal information in both directions. Our proposed approach outperforms other state-of-the-art methods on the benchmark dataset.