mlsanityData to 7 Oct 2026

Towards Motion-aware Referring Image Segmentation

arXiv 2603.17413 ↗First posted 18 Mar 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 76 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

41 of 76 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  9. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
29 more
  1. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  2. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  3. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  4. 2016Segmentation from Natural Language Expressionscited by 257
  5. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  7. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  8. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  9. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  10. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  11. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  12. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  13. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  14. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  15. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  16. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  17. 2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
  18. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  19. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  20. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  21. 2022CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentationcited by 28
  22. 2025Densely Connected Parameter-Efficient Tuning for Referring Image Segmentationcited by 26
  23. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  24. 2024Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentationcited by 21
  25. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  26. 2021Contrastive Fine-tuning Improves Robustness for Neural Rankerscited by 12
  27. 2024Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understandingcited by 7
  28. 2024Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentationcited by 7
  29. 2025Latent Expression Generation for Referring Image Segmentation and Groundingcited by 6

Built on by

Not cited yet.