# Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions

arXiv: https://arxiv.org/abs/2005.01655 · HTML: https://mlsanity.com/paper/2005.01655/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 30 of 33)

- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers](https://mlsanity.com/paper/2102.00529/) (2021)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [Measuring Progress in Fine-grained Vision-and-Language Understanding](https://mlsanity.com/paper/2305.07558/) (2023)
- [CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models](https://mlsanity.com/paper/2109.01401/) (2021)
- [FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension](https://mlsanity.com/paper/2409.14750/) (2024)
- [How and where does CLIP process negation?](https://mlsanity.com/paper/2407.10488/) (2024)
- [If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions](https://mlsanity.com/paper/2403.16442/) (2024)
- [Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships](https://mlsanity.com/paper/2203.14260/) (2022)
- [Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?](https://mlsanity.com/paper/2210.12079/) (2022)
- [NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language](https://mlsanity.com/paper/2509.25757/) (2025)
- [New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration](https://mlsanity.com/paper/2502.20104/) (2025)
- [Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data](https://mlsanity.com/paper/2111.09190/) (2021)
- [ULN: Towards Underspecified Vision-and-Language Navigation](https://mlsanity.com/paper/2210.10020/) (2022)
- [Revisiting Multi-Modal LLM Evaluation](https://mlsanity.com/paper/2408.05334/) (2024)
- [MOSS-VL Technical Report](https://mlsanity.com/paper/2608.15045/) (2026)
- [Attention cannot be an Explanation](https://mlsanity.com/paper/2201.11194/) (2022)
- [Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding](https://mlsanity.com/paper/2201.09639/) (2022)
- [VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching](https://mlsanity.com/paper/2105.05636/) (2021)
- [Discourse Analysis for Evaluating Coherence in Video Paragraph Captions](https://mlsanity.com/paper/2201.06207/) (2022)
- [Scalable Performance Analysis for Vision-Language Models](https://mlsanity.com/paper/2305.18786/) (2023)
- [Neural Variational Learning for Grounded Language Acquisition](https://mlsanity.com/paper/2107.14593/) (2021)
- [VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension](https://mlsanity.com/paper/2601.12781/) (2026)

## Abstract (arXiv)

Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn't matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn't. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are 12% to 23% lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of-the-art model for this task. Our datasets are publicly available at https://github.com/aws/aws-refcocog-adv