mlsanityData to 7 Oct 2026

SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation

arXiv 2101.08833 ↗First posted 21 Jan 2021cs.CVcited by 106

Not on the map. Why?

Abstract (arXiv)

In this paper we introduce a Transformer-based approach to video object segmentation (VOS). To address compounding error and scalability issues of prior work, we propose a scalable, end-to-end method for VOS called Sparse Spatiotemporal Transformers (SST). SST extracts per-pixel representations for each object in a video using sparse attention over spatiotemporal features. Our attention-based formulation for VOS allows a model to learn to attend over a history of multiple frames and provides suitable inductive bias for performing correspondence-like computations necessary for solving motion segmentation. We demonstrate the effectiveness of attention-based over recurrent networks in the spatiotemporal domain. Our method achieves competitive results on YouTube-VOS and DAVIS 2017 with improved scalability and robustness to occlusions compared with the state of the art. Code is available at https://github.com/dukebw/SSTVOS.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

45 of 54 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
33 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  4. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  5. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  6. 2018Video Action Transformer Networkcited by 316
  7. 2020Exploring Self-attention for Image Recognitioncited by 313
  8. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  9. 2016One-Shot Video Object Segmentationcited by 233
  10. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  11. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  12. 2016Learning Video Object Segmentation from Static Imagescited by 151
  13. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  14. 2020Kernelized Memory Network for Video Object Segmentationcited by 111
  15. 2017Online Adaptation of Convolutional Neural Networks for Video Object Segmentationcited by 109
  16. 2018Efficient Video Object Segmentation via Network Modulationcited by 98
  17. 2017Video Object Segmentation Without Temporal Informationcited by 93
  18. 2017SegFlow: Joint Learning for Video Object Segmentation and Optical Flowcited by 92
  19. 2018VideoMatch: Matching based Video Object Segmentationcited by 90
  20. 2019RVOS: End-to-End Recurrent Network for Video Object Segmentationcited by 88
  21. 2018PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentationcited by 81
  22. 2017FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videoscited by 77
  23. 2017Learning Video Object Segmentation with Visual Memorycited by 66
  24. 2018A Generative Appearance Model for End-to-end Video Object Segmentationcited by 66
  25. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  26. 2018Fast and Accurate Online Video Object Segmentation via Tracking Partscited by 57
  27. 2018Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagationcited by 56
  28. 2018CNN in MRF: Video Object Segmentation via Inference in A CNN-Based Higher-Order Spatio-Temporal MRFcited by 48
  29. 2018MaskRNN: Instance Level Video Object Segmentationcited by 48
  30. 2016Video Propagation Networkscited by 45
  31. 2014Do Convnets Learn Correspondence?cited by 28
  32. 2017Recurrent Neural Networks for Semantic Instance Segmentationcited by 11
  33. 2019BoLTVOS: Box-Level Tracking for Video Object Segmentationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 30; 2023: 34; 2024: 18; 2025: 7; 2026: 2.

106 papers.

Sort:

By area:

  1. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  2. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  3. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  4. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  5. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  6. 2023Putting the Object Back into Video Object Segmentationcited by 182
  7. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  8. 2022Audio-Visual Segmentationcited by 148
  9. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  10. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  11. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  12. 2021VidTr: Video Transformer Without Convolutionscited by 112
38 more
  1. 2021Full-Duplex Strategy for Video Object Segmentationcited by 78
  2. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 77
  3. 2023Audio-Visual Segmentation with Semanticscited by 73
  4. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
  5. 2022Video Transformers: A Surveycited by 63
  6. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  7. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  8. 2024Language Repository for Long Video Understandingcited by 55
  9. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  10. 2022Recurrent Dynamic Embedding for Video Object Segmentationcited by 52
  11. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  12. 2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50
  13. 2022Breaking the "Object" in Video Object Segmentationcited by 48
  14. 2023Annotation-free Audio-Visual Segmentationcited by 44
  15. 2022Delving into Sequential Patches for Deepfake Detectioncited by 41
  16. 2022Per-Clip Video Object Segmentationcited by 40
  17. 2022Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentationcited by 37
  18. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  19. 2021Reliable Propagation-Correction Modulation for Video Object Segmentationcited by 36
  20. 2023Improving Audio-Visual Segmentation with Bidirectional Generationcited by 36
  21. 2022Learning Quality-aware Dynamic Memory for Video Object Segmentationcited by 35
  22. 2022Look Before You Match: Instance Understanding Matters in Video Object Segmentationcited by 33
  23. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 32
  24. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 30
  25. 2023Scalable Video Object Segmentation with Simplified Frameworkcited by 29
  26. 2023Discovering Sounding Objects by Audio Queries for Audio Visual Segmentationcited by 27
  27. 2022SWEM: Towards Real-Time Video Object Segmentation with Sequential Weighted Expectation-Maximizationcited by 26
  28. 2024Learning Natural Consistency Representation for Face Forgery Video Detectioncited by 26
  29. 2023Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentationcited by 25
  30. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  31. 2022Global Spectral Filter Memory Network for Video Object Segmentationcited by 22
  32. 2023DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Taskscited by 22
  33. 2024RMem: Restricted Memory Banks Improve Video Object Segmentationcited by 22
  34. 2023Boosting Video Object Segmentation via Space-time Correspondence Learningcited by 21
  35. 2025EdgeTAM: On-Device Track Anything Modelcited by 21
  36. 2021Can An Image Classifier Suffice For Action Recognition?cited by 20
  37. 2024Efficient Track Anythingcited by 20
  38. 2021VisQA: X-raying Vision and Language Reasoning in Transformerscited by 19

50 most cited of 106 All 106 (JSON)