mlsanityData to 7 Oct 2026

Temporally Consistent Referring Video Object Segmentation with Hybrid Memory

arXiv 2403.19407 ↗First posted 28 Mar 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Referring Video Object Segmentation (R-VOS) methods face challenges in maintaining consistent object segmentation due to temporal context variability and the presence of other visually similar objects. We propose an end-to-end R-VOS paradigm that explicitly models temporal instance consistency alongside the referring segmentation. Specifically, we introduce a novel hybrid memory that facilitates inter-frame collaboration for robust spatio-temporal matching and propagation. Features of frames with automatically generated high-quality reference masks are propagated to segment the remaining frames based on multi-granularity association to achieve temporally consistent R-VOS. Furthermore, we propose a new Mask Consistency Score (MCS) metric to evaluate the temporal consistency of video segmentation. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin, leading to top-ranked performance on popular R-VOS benchmarks, i.e., Ref-YouTube-VOS (67.1%) and Ref-DAVIS17 (65.6%). The code is available at https://github.com/bo-miao/HTR.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 71 of its 107 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

71 of 107 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2021Video Swin Transformercited by 1,090
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
38 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  4. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  5. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  6. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  7. 2020Conditional Convolutions for Instance Segmentationcited by 314
  8. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  9. 2016Segmentation from Natural Language Expressionscited by 257
  10. 2016One-Shot Video Object Segmentationcited by 233
  11. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  12. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  13. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  14. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  15. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  16. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  17. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  18. 2021Mask2Former for Video Instance Segmentationcited by 169
  19. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  20. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  21. 2016Learning Video Object Segmentation from Static Imagescited by 151
  22. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  23. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  24. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  25. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  26. 2020CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinementcited by 113
  27. 2020Kernelized Memory Network for Video Object Segmentationcited by 111
  28. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  29. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  30. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  31. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  32. 2020Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinementcited by 101
  33. 2018Efficient Video Object Segmentation via Network Modulationcited by 98
  34. 2017Video Object Segmentation Without Temporal Informationcited by 93
  35. 2019Towards High-Resolution Salient Object Detectioncited by 92
  36. 2018PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentationcited by 81
  37. 2021Learning Position and Target Consistency for Memory-based Video Object Segmentationcited by 63
  38. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  2. 2025Multimodal Referring Segmentation: A Surveycited by 22
  3. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  4. 2024Referring Human Pose and Mask Estimation in the Wildcited by 6
  5. 2024Referring Video Object Segmentation via Language-aligned Track Selectioncited by 3
  6. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigationcited by 2
  7. 2024Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentationcited by 1
  8. 2026The Essence of Balance for Self-Improving Agents in Vision-and-Language Navigationcited by 1
  9. 2025Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videoscited by 0
  10. 2025Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrievalcited by 0
  11. 2025LTCA: Long-range Temporal Context Attention for Referring Video Object Segmentationcited by 0
  12. 2026Show Me When and Where: Towards Referring Video Object Segmentation in the Wildcited by 0
2 more
  1. 2026Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioningcited by 0
  2. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentationcited by 0