# Recurrent Dynamic Embedding for Video Object Segmentation

arXiv: https://arxiv.org/abs/2205.03761 · HTML: https://mlsanity.com/paper/2205.03761/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 8 May 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation](https://mlsanity.com/paper/1907.12347/) (2019)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [Collaborative Video Object Segmentation by Foreground-Background Integration](https://mlsanity.com/paper/2003.08333/) (2020)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement](https://mlsanity.com/paper/2005.02551/) (2020)
- [Kernelized Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2007.08270/) (2020)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [Efficient Video Object Segmentation via Network Modulation](https://mlsanity.com/paper/1802.01218/) (2018)
- [Video Object Segmentation Without Temporal Information](https://mlsanity.com/paper/1709.06031/) (2017)
- [Towards High-Resolution Salient Object Detection](https://mlsanity.com/paper/1908.07274/) (2019)
- [VideoMatch: Matching based Video Object Segmentation](https://mlsanity.com/paper/1809.01123/) (2018)
- [Efficient Regional Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2103.12934/) (2021)
- [PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentation](https://mlsanity.com/paper/1807.09190/) (2018)
- [Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration](https://mlsanity.com/paper/2010.06349/) (2020)
- [Hierarchical Memory Matching Network for Video Object Segmentation](https://mlsanity.com/paper/2109.11404/) (2021)
- [SwiftNet: Real-time Video Object Segmentation](https://mlsanity.com/paper/2102.04604/) (2021)

## Built on by (most cited first, 30 of 52)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [LVOS: A Benchmark for Long-term Video Object Segmentation](https://mlsanity.com/paper/2211.10181/) (2022)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [XMem++: Production-level Video Segmentation From Few Annotated Frames](https://mlsanity.com/paper/2307.15958/) (2023)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [Look Before You Match: Instance Understanding Matters in Video Object Segmentation](https://mlsanity.com/paper/2212.06826/) (2022)
- [Advancing Complex Video Object Segmentation via Progressive Concept Construction](https://mlsanity.com/paper/2507.15852/) (2025)
- [UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces](https://mlsanity.com/paper/2312.15715/) (2023)
- [Event-assisted Low-Light Video Object Segmentation](https://mlsanity.com/paper/2404.01945/) (2024)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [Efficient Track Anything](https://mlsanity.com/paper/2411.18933/) (2024)
- [EdgeTAM: On-Device Track Anything Model](https://mlsanity.com/paper/2501.07256/) (2025)
- [PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation](https://mlsanity.com/paper/2309.12303/) (2023)
- [MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation](https://mlsanity.com/paper/2303.07815/) (2023)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection](https://mlsanity.com/paper/2403.01968/) (2024)
- [Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment](https://mlsanity.com/paper/2506.14271/) (2025)
- [Two-shot Video Object Segmentation](https://mlsanity.com/paper/2303.12078/) (2023)
- [Learning Spatial-Semantic Features for Robust Video Object Segmentation](https://mlsanity.com/paper/2407.07760/) (2024)
- [LiVOS: Light Video Object Segmentation with Gated Linear Matching](https://mlsanity.com/paper/2411.02818/) (2024)
- [Learning to Learn Better for Video Object Segmentation](https://mlsanity.com/paper/2212.02112/) (2022)
- [EGVD: Event-Guided Video Deraining](https://mlsanity.com/paper/2309.17239/) (2023)

## Abstract (arXiv)

Space-time memory (STM) based video object segmentation (VOS) networks usually keep increasing memory bank every several frames, which shows excellent performance. However, 1) the hardware cannot withstand the ever-increasing memory requirements as the video length increases. 2) Storing lots of information inevitably introduces lots of noise, which is not conducive to reading the most important information from the memory bank. In this paper, we propose a Recurrent Dynamic Embedding (RDE) to build a memory bank of constant size. Specifically, we explicitly generate and update RDE by the proposed Spatio-temporal Aggregation Module (SAM), which exploits the cue of historical information. To avoid error accumulation owing to the recurrent usage of SAM, we propose an unbiased guidance loss during the training stage, which makes SAM more robust in long videos. Moreover, the predicted masks in the memory bank are inaccurate due to the inaccurate network inference, which affects the segmentation of the query frame. To address this problem, we design a novel self-correction strategy so that the network can repair the embeddings of masks with different qualities in the memory bank. Extensive experiments show our method achieves the best tradeoff between performance and speed. Code is available at https://github.com/Limingxing00/RDE-VOS-CVPR2022.