mlsanityData to 7 Oct 2026

Grounded Image Text Matching with Mismatched Relation Reasoning

arXiv 2308.01236 ↗First posted 2 Aug 2023cs.CV cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper introduces Grounded Image Text Matching with Mismatched Relation (GITM-MR), a novel visual-linguistic joint task that evaluates the relation understanding capabilities of transformer-based pre-trained models. GITM-MR requires a model to first determine if an expression describes an image, then localize referred objects or ground the mismatched parts of the text. We provide a benchmark for evaluating pre-trained models on this task, with a focus on the challenging settings of limited data and out-of-distribution sentence lengths. Our evaluation demonstrates that pre-trained models lack data efficiency and length generalization ability. To address this, we propose the Relation-sensitive Correspondence Reasoning Network (RCRN), which incorporates relation-aware reasoning via bi-directional message propagation guided by language structure. RCRN can be interpreted as a modular program and delivers strong performance in both length generalization and data efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

55 of 67 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2017Graph Attention Networkscited by 8,854
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  10. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  11. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
38 more
  1. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  2. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  3. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  5. 2022Exploring Length Generalization in Large Language Modelscited by 204
  6. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  7. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  8. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  9. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  10. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  11. 2018Invariance, Causality and Robustnesscited by 123
  12. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  13. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  14. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  15. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  16. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  17. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  18. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  19. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  20. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  21. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  22. 2019Stable Learning via Sample Reweightingcited by 70
  23. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  24. 2020Stable Prediction with Model Misspecification and Agnostic Distribution Shiftcited by 62
  25. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  26. 2017Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queriescited by 56
  27. 2021Look Before You Leap: Learning Landmark Features for One-Stage Visual Groundingcited by 51
  28. 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
  29. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 41
  30. 2018A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labelingcited by 37
  31. 2020Unshuffling Data for Improved Generalizationcited by 35
  32. 2021Sequence Length is a Domain: Length-based Overfitting in Transformer Modelscited by 34
  33. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  34. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  35. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  36. 2019Saliency-Guided Attention Network for Image-Sentence Matchingcited by 21
  37. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  38. 2020Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matchingcited by 15

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 1.

8 papers.

  1. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  2. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 9
  3. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  4. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  5. 2025Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrievalcited by 4
  6. 2025Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipationcited by 1
  7. 2025EventRR: Event Referential Reasoning for Referring Video Object Segmentationcited by 0
  8. 2026DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Datasetcited by 0