mlsanityData to 7 Oct 2026

Language as Queries for Referring Video Object Segmentation

arXiv 2201.00487 ↗First posted 3 Jan 2022cs.CVcited by 180

Not on the map. Why?

Abstract (arXiv)

Referring video object segmentation (R-VOS) is an emerging cross-modal task that aims to segment the target object referred by a language expression in all video frames. In this work, we propose a simple and unified framework built upon Transformer, termed ReferFormer. It views the language as queries and directly attends to the most relevant regions in the video frames. Concretely, we introduce a small set of object queries conditioned on the language as the input to the Transformer. In this manner, all the queries are obligated to find the referred objects only. They are eventually transformed into dynamic kernels which capture the crucial object-level information, and play the role of convolution filters to generate the segmentation masks from feature maps. The object tracking is achieved naturally by linking the corresponding queries across frames. This mechanism greatly simplifies the pipeline and the end-to-end framework is significantly different from the previous methods. Extensive experiments on Ref-Youtube-VOS, Ref-DAVIS17, A2D-Sentences and JHMDB-Sentences show the effectiveness of ReferFormer. On Ref-Youtube-VOS, Refer-Former achieves 55.6J&F with a ResNet-50 backbone without bells and whistles, which exceeds the previous state-of-the-art performance by 8.4 points. In addition, with the strong Swin-Large backbone, ReferFormer achieves the best J&F of 64.2 among all existing methods. Moreover, we show the impressive results of 55.0 mAP and 43.7 mAP on A2D-Sentences andJHMDB-Sentences respectively, which significantly outperforms the previous methods by a large margin. Code is publicly available at https://github.com/wjn922/ReferFormer.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 53 of its 62 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

53 of 62 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
38 more
  1. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  2. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  3. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  4. 2021Video Swin Transformercited by 1,090
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2021Multiscale Vision Transformerscited by 896
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  10. 2017Focal Loss for Dense Object Detectioncited by 749
  11. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  12. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  13. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  14. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  15. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  16. 2021Transformer Trackingcited by 412
  17. 2019Video Instance Segmentationcited by 391
  18. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  19. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  20. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  21. 2020Conditional Convolutions for Instance Segmentationcited by 314
  22. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  23. 2016Segmentation from Natural Language Expressionscited by 257
  24. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  25. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  26. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  27. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  28. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  29. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  30. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  31. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  32. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  33. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  34. 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105
  35. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  36. 2021Video Instance Segmentation with a Propose-Reduce Paradigmcited by 74
  37. 2021Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentationcited by 42
  38. 2021Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Networkcited by 28

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 39; 2024: 40; 2025: 55; 2026: 34.

180 papers.

Sort:

By area:

  1. 2023Segment Everything Everywhere All at Oncecited by 546
  2. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  3. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  5. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  6. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  7. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  8. 2022DETRs with Hybrid Matchingcited by 149
  9. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  10. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  11. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  12. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
38 more
  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  2. 2023Referring Multi-Object Trackingcited by 91
  3. 2023Audio-Visual Segmentation with Semanticscited by 73
  4. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  5. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  6. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  7. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  8. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  9. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  10. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  11. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  12. 2023Visual In-Context Promptingcited by 52
  13. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  14. 2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
  15. 2022Towards Robust Referring Image Segmentationcited by 49
  16. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  17. 2023Annotation-free Audio-Visual Segmentationcited by 44
  18. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  19. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  20. 2024LVOS: A Benchmark for Large-scale Long-term Video Object Segmentationcited by 41
  21. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  22. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  23. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  24. 2023Tracking with Human-Intent Reasoningcited by 32
  25. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  26. 2023iKUN: Speak to Trackers without Retrainingcited by 31
  27. 2025ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundationscited by 30
  28. 2023Universal Segmentation at Arbitrary Granularity with Language Instructioncited by 29
  29. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  30. 2023Temporal Collection and Distribution for Referring Video Object Segmentationcited by 27
  31. 2023Discovering Sounding Objects by Audio Queries for Audio Visual Segmentationcited by 27
  32. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
  33. 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 26
  34. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  35. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  36. 2023RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentationcited by 24
  37. 2023LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentationcited by 23
  38. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22

50 most cited of 180 All 180 (JSON)