# Temporal Memory Attention for Video Semantic Segmentation

arXiv: https://arxiv.org/abs/2102.08643 · HTML: https://mlsanity.com/paper/2102.08643/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 23 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [Temporally Distributed Networks for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2004.01800/) (2020)
- [Semantic Video Segmentation by Gated Recurrent Flow Propagation](https://mlsanity.com/paper/1612.08871/) (2016)
- [Semantic Video CNNs through Representation Warping](https://mlsanity.com/paper/1708.03088/) (2017)
- [Fast Video Object Segmentation using the Global Context Module](https://mlsanity.com/paper/2001.11243/) (2020)
- [Efficient Semantic Video Segmentation with Per-frame Inference](https://mlsanity.com/paper/2002.11433/) (2020)
- [Low-Latency Video Semantic Segmentation](https://mlsanity.com/paper/1804.00389/) (2018)
- [Clockwork Convnets for Video Semantic Segmentation](https://mlsanity.com/paper/1608.03609/) (2016)
- [Dynamic Video Segmentation Network](https://mlsanity.com/paper/1804.00931/) (2018)

## Built on by (most cited first, 30 of 50)

- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [Stubborn: A Strong Baseline for Indoor Object Navigation](https://mlsanity.com/paper/2203.07359/) (2022)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [X-SAM: From Segment Anything to Any Segmentation](https://mlsanity.com/paper/2508.04655/) (2025)
- [OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion](https://mlsanity.com/paper/2407.07844/) (2024)
- [3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model](https://mlsanity.com/paper/2505.22657/) (2025)
- [Spatio-Temporal Self-Attention Network for Video Saliency Prediction](https://mlsanity.com/paper/2108.10696/) (2021)
- [LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Benchmark](https://mlsanity.com/paper/2308.09618/) (2023)
- [Exploiting Temporal State Space Sharing for Video Semantic Segmentation](https://mlsanity.com/paper/2503.20824/) (2025)
- [Event-guided Low-light Video Semantic Segmentation](https://mlsanity.com/paper/2411.00639/) (2024)
- [TFNet: Exploiting Temporal Cues for Fast and Accurate LiDAR Semantic Segmentation](https://mlsanity.com/paper/2309.07849/) (2023)
- [Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2401.15261/) (2024)
- [Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models](https://mlsanity.com/paper/2405.16947/) (2024)
- [Temporal Context for Robust Maritime Obstacle Detection](https://mlsanity.com/paper/2203.05352/) (2022)
- [SMITE: Segment Me In TimE](https://mlsanity.com/paper/2410.18538/) (2024)
- [NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation](https://mlsanity.com/paper/2307.08695/) (2023)
- [Scene-aware Learning Network for Radar Object Detection](https://mlsanity.com/paper/2107.01469/) (2021)
- [Temporally stable video segmentation without video annotations](https://mlsanity.com/paper/2110.08893/) (2021)
- [Deep Common Feature Mining for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2403.02689/) (2024)
- [Memory Based Video Scene Parsing](https://mlsanity.com/paper/2109.00373/) (2021)
- [Motion-state Alignment for Video Semantic Segmentation](https://mlsanity.com/paper/2304.08820/) (2023)
- [Interactive Segmentation for Diverse Gesture Types Without Context](https://mlsanity.com/paper/2307.10518/) (2023)
- [Force Sensing Guided Artery-Vein Segmentation via Sequential Ultrasound Images](https://mlsanity.com/paper/2407.21394/) (2024)
- [Online Dense Point Tracking with Streaming Memory](https://mlsanity.com/paper/2503.06471/) (2025)
- [High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous Flight](https://mlsanity.com/paper/2503.15676/) (2025)
- [Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation](https://mlsanity.com/paper/2604.10950/) (2026)
- [Semantic Segmentation on VSPW Dataset through Aggregation of Transformer Models](https://mlsanity.com/paper/2109.01316/) (2021)
- [Learning to Adapt to Online Streams with Distribution Shifts](https://mlsanity.com/paper/2303.01630/) (2023)

## Abstract (arXiv)

Video semantic segmentation requires to utilize the complex temporal relations between frames of the video sequence. Previous works usually exploit accurate optical flow to leverage the temporal relations, which suffer much from heavy computational cost. In this paper, we propose a Temporal Memory Attention Network (TMANet) to adaptively integrate the long-range temporal relations over the video sequence based on the self-attention mechanism without exhaustive optical flow prediction. Specially, we construct a memory using several past frames to store the temporal information of the current frame. We then propose a temporal memory attention module to capture the relation between the current frame and the memory to enhance the representation of the current frame. Our method achieves new state-of-the-art performances on two challenging video semantic segmentation datasets, particularly 80.3% mIoU on Cityscapes and 76.5% mIoU on CamVid with ResNet-50.