# Meta Compositional Referring Expression Segmentation

arXiv: https://arxiv.org/abs/2304.04415 · HTML: https://mlsanity.com/paper/2304.04415/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [Referring Image Segmentation via Cross-Modal Progressive Comprehension](https://mlsanity.com/paper/2010.00514/) (2020)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)
- [Linguistic Structure Guided Context Modeling for Referring Image Segmentation](https://mlsanity.com/paper/2010.00515/) (2020)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [Dynamic Multimodal Instance Segmentation guided by natural language queries](https://mlsanity.com/paper/1807.02257/) (2018)
- [Learning Meta Face Recognition in Unseen Domains](https://mlsanity.com/paper/2003.07733/) (2020)
- [Referring Expression Object Segmentation with Caption-Aware Consistency](https://mlsanity.com/paper/1910.04748/) (2019)

## Built on by (most cited first, 18 of 18)

- [Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding](https://mlsanity.com/paper/2503.06287/) (2025)
- [Mask Grounding for Referring Image Segmentation](https://mlsanity.com/paper/2312.12198/) (2023)
- [Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models](https://mlsanity.com/paper/2308.16777/) (2023)
- [Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers](https://mlsanity.com/paper/2407.08394/) (2024)
- [Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation](https://mlsanity.com/paper/2404.11998/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Learning to Unlearn for Robust Machine Unlearning](https://mlsanity.com/paper/2407.10494/) (2024)
- [Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation](https://mlsanity.com/paper/2504.00356/) (2025)
- [Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentation](https://mlsanity.com/paper/2407.07412/) (2024)
- [RESMatch: Referring Expression Segmentation in a Semi-Supervised Manner](https://mlsanity.com/paper/2402.05589/) (2024)
- [Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2412.07183/) (2024)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [Deep Instruction Tuning for Segment Anything Model](https://mlsanity.com/paper/2404.00650/) (2024)
- [LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentation](https://mlsanity.com/paper/2504.14467/) (2025)
- [Synchronizing Vision and Language: Bidirectional Token-Masking AutoEncoder for Referring Image Segmentation](https://mlsanity.com/paper/2311.17952/) (2023)
- [Spatial Semantic Recurrent Mining for Referring Image Segmentation](https://mlsanity.com/paper/2405.09006/) (2024)
- [Rethinking Referring Object Removal](https://mlsanity.com/paper/2403.09128/) (2024)
- [HARIS: Human-Like Attention for Reference Image Segmentation](https://mlsanity.com/paper/2405.10707/) (2024)

## Abstract (arXiv)

Referring expression segmentation aims to segment an object described by a language expression from an image. Despite the recent progress on this task, existing models tackling this task may not be able to fully capture semantics and visual representations of individual concepts, which limits their generalization capability, especially when handling novel compositions of learned concepts. In this work, through the lens of meta learning, we propose a Meta Compositional Referring Expression Segmentation (MCRES) framework to enhance model compositional generalization performance. Specifically, to handle various levels of novel compositions, our framework first uses training data to construct a virtual training set and multiple virtual testing sets, where data samples in each virtual testing set contain a level of novel compositions w.r.t. the virtual training set. Then, following a novel meta optimization scheme to optimize the model to obtain good testing performance on the virtual testing sets after training on the virtual training set, our framework can effectively drive the model to better capture semantics and visual representations of individual concepts, and thus obtain robust generalization performance even when handling novel compositions. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our framework.