# TransVOS: Video Object Segmentation with Transformers

arXiv: https://arxiv.org/abs/2106.00588 · HTML: https://mlsanity.com/paper/2106.00588/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [Collaborative Video Object Segmentation by Foreground-Background Integration](https://mlsanity.com/paper/2003.08333/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Kernelized Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2007.08270/) (2020)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [VideoMatch: Matching based Video Object Segmentation](https://mlsanity.com/paper/1809.01123/) (2018)
- [Efficient Regional Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2103.12934/) (2021)
- [PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentation](https://mlsanity.com/paper/1807.09190/) (2018)
- [Learning Fast and Robust Target Models for Video Object Segmentation](https://mlsanity.com/paper/2003.00908/) (2020)
- [SwiftNet: Real-time Video Object Segmentation](https://mlsanity.com/paper/2102.04604/) (2021)
- [A Generative Appearance Model for End-to-end Video Object Segmentation](https://mlsanity.com/paper/1811.11611/) (2018)

## Built on by (most cited first, 15 of 15)

- [A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection](https://mlsanity.com/paper/2203.04708/) (2022)
- [Per-Clip Video Object Segmentation](https://mlsanity.com/paper/2208.01924/) (2022)
- [BATMAN: Bilateral Attention Transformer in Motion-Appearance Neighboring Space for Video Object Segmentation](https://mlsanity.com/paper/2208.01159/) (2022)
- [PSNet: Parallel Symmetric Network for Video Salient Object Detection](https://mlsanity.com/paper/2210.05912/) (2022)
- [Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection](https://mlsanity.com/paper/2403.01968/) (2024)
- [Learning Referring Video Object Segmentation from Weak Annotation](https://mlsanity.com/paper/2308.02162/) (2023)
- [Efficient Video Object Segmentation via Modulated Cross-Attention Memory](https://mlsanity.com/paper/2403.17937/) (2024)
- [Hierarchical Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2307.08263/) (2023)
- [UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection](https://mlsanity.com/paper/2309.08220/) (2023)
- [CamoSAM2: SAM2-oriented Prompt Auto-Refinement for Video Camouflaged Object Detection](https://mlsanity.com/paper/2504.00375/) (2025)
- [FlowVOS: Weakly-Supervised Visual Warping for Detail-Preserving and Temporally Consistent Single-Shot Video Object Segmentation](https://mlsanity.com/paper/2111.10621/) (2021)
- [Two-Stream Networks for Object Segmentation in Videos](https://mlsanity.com/paper/2208.04026/) (2022)
- [Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation](https://mlsanity.com/paper/2308.13505/) (2023)
- [DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queries](https://mlsanity.com/paper/2408.15813/) (2024)
- [Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation](https://mlsanity.com/paper/2409.14343/) (2024)

## Abstract (arXiv)

Recently, Space-Time Memory Network (STM) based methods have achieved state-of-the-art performance in semi-supervised video object segmentation (VOS). A crucial problem in this task is how to model the dependency both among different frames and inside every frame. However, most of these methods neglect the spatial relationships (inside each frame) and do not make full use of the temporal relationships (among different frames). In this paper, we propose a new transformer-based framework, termed TransVOS, introducing a vision transformer to fully exploit and model both the temporal and spatial relationships. Moreover, most STM-based approaches employ two separate encoders to extract features of two significant inputs, i.e., reference sets (history frames with predicted masks) and query frame (current frame), respectively, increasing the models' parameters and complexity. To slim the popular two-encoder pipeline while keeping the effectiveness, we design a single two-path feature extractor to encode the above two inputs in a unified way. Extensive experiments demonstrate the superiority of our TransVOS over state-of-the-art methods on both DAVIS and YouTube-VOS datasets.