mlsanityData to 7 Oct 2026

A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation

arXiv 2211.07919 ↗First posted 15 Nov 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Reference Expression Segmentation (RES) and Reference Expression Generation (REG) are mutually inverse tasks that can be naturally jointly trained. Though recent work has explored such joint training, the mechanism of how RES and REG can benefit each other is still unclear. In this paper, we propose a unified mutual supervision framework that enables two tasks to improve each other. Our mutual supervision contains two directions. On the one hand, Disambiguation Supervision leverages the expression unambiguity measurement provided by RES to enhance the language generation of REG. On the other hand, Generation Supervision uses expressions automatically generated by REG to scale up the training of RES. Such unified mutual supervision effectively improves two tasks by solving their bottleneck problems. Extensive experiments show that our approach significantly outperforms all existing methods on REG and RES tasks under the same setting, and detailed ablation studies demonstrate the effectiveness of all components in our framework.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

46 of 67 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
34 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  2. 2016Self-critical Sequence Training for Image Captioningcited by 544
  3. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  4. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  5. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  6. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  7. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  8. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  9. 2016Segmentation from Natural Language Expressionscited by 257
  10. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  11. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  13. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  14. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  15. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  16. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  17. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  18. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  19. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  20. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  21. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  22. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  23. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  24. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  25. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  26. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  27. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  28. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  29. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  30. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  31. 2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
  32. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33
  33. 2019Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressionscited by 12
  34. 2022Towards Unifying Reference Expression Generation and Comprehensioncited by 6

Built on by

Cited by year

Citing papers per year: 2023: 5.

5 papers.

  1. 2023Segment Everything Everywhere All at Oncecited by 546
  2. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  3. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  4. 2023Advancing Referring Expression Segmentation Beyond Single Imagecited by 16
  5. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8