mlsanityData to 7 Oct 2026

Hierarchical Spatiotemporal Transformers for Video Object Segmentation

arXiv 2307.08263 ↗First posted 17 Jul 2023cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel framework called HST for semi-supervised video object segmentation (VOS). HST extracts image and video features using the latest Swin Transformer and Video Swin Transformer to inherit their inductive bias for the spatiotemporal locality, which is essential for temporally coherent VOS. To take full advantage of the image and video features, HST casts image and video features as a query and memory, respectively. By applying efficient memory read operations at multiple scales, HST produces hierarchical features for the precise reconstruction of object masks. HST shows effectiveness and robustness in handling challenging scenarios with occluded and fast-moving objects under cluttered backgrounds. In particular, HST-B outperforms the state-of-the-art competitors on multiple popular benchmarks, i.e., YouTube-VOS (85.0%), DAVIS 2017 (85.9%), and DAVIS 2016 (94.0%).

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

49 of 63 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
37 more
  1. 2021Video Swin Transformercited by 1,090
  2. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  3. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  4. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  5. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  6. 2016One-Shot Video Object Segmentationcited by 233
  7. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  8. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  9. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  10. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  11. 2016Learning Video Object Segmentation from Static Imagescited by 151
  12. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  13. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  14. 2020Video Object Segmentation with Episodic Graph Memory Networkscited by 118
  15. 2020Kernelized Memory Network for Video Object Segmentationcited by 111
  16. 2017Online Adaptation of Convolutional Neural Networks for Video Object Segmentationcited by 109
  17. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  18. 2020Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinementcited by 101
  19. 2018Efficient Video Object Segmentation via Network Modulationcited by 98
  20. 2018VideoMatch: Matching based Video Object Segmentationcited by 90
  21. 2014Salient Object Detection: A Discriminative Regional Feature Integration Approachcited by 89
  22. 2021Efficient Regional Memory Network for Video Object Segmentationcited by 86
  23. 2018PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentationcited by 81
  24. 2020Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integrationcited by 81
  25. 2021Hierarchical Memory Matching Network for Video Object Segmentationcited by 72
  26. 2021SwiftNet: Real-time Video Object Segmentationcited by 67
  27. 2020Fast Video Object Segmentation using the Global Context Modulecited by 65
  28. 2021Learning Position and Target Consistency for Memory-based Video Object Segmentationcited by 63
  29. 2019RANet: Ranking Attention Network for Fast Video Object Segmentationcited by 60
  30. 2019Video Object Segmentation and Tracking: A Surveycited by 49
  31. 2018MaskRNN: Instance Level Video Object Segmentationcited by 48
  32. 2022Per-Clip Video Object Segmentationcited by 40
  33. 2020Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matchingcited by 37
  34. 2022Learning Quality-aware Dynamic Memory for Video Object Segmentationcited by 35
  35. 2021TransVOS: Video Object Segmentation with Transformerscited by 15
  36. 2021Siamese Network with Interactive Transformer for Video Object Segmentationcited by 11
  37. 2022Towards Robust Video Object Segmentation with Adaptive Object Calibrationcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024DiDA: Video Object Segmentation with Distillation Learning of Deformable Attentioncited by 2