mlsanityData to 14 Oct 2026

How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding

arXiv 2402.19116 ↗First posted 29 Feb 2024cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Weakly-supervised Phrase Grounding (WPG) is an emerging task of inferring the fine-grained phrase-region matching, while merely leveraging the coarse-grained sentence-image pairs for training. However, existing studies on WPG largely ignore the implicit phrase-region matching relations, which are crucial for evaluating the capability of models in understanding the deep multimodal semantics. To this end, this paper proposes an Implicit-Enhanced Causal Inference (IECI) approach to address the challenges of modeling the implicit relations and highlighting them beyond the explicit. Specifically, this approach leverages both the intervention and counterfactual techniques to tackle the above two challenges respectively. Furthermore, a high-quality implicit-enhanced dataset is annotated to evaluate IECI and detailed evaluations show the great advantages of IECI over the state-of-the-art baselines. Particularly, we observe an interesting finding that IECI outperforms the advanced multimodal LLMs by a large margin on this implicit-enhanced dataset, which may facilitate more research to evaluate the multimodal LLMs in this direction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 51 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

37 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021Causal Attention for Vision-Language Tasks5 Mar

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  10. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  11. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  12. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
23 more
  1. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  2. 2020Causal Intervention for Weakly-Supervised Semantic Segmentationcited by 229
  3. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  4. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  5. 2020Visual Commonsense R-CNNcited by 106
  6. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  7. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  8. 2018Unsupervised Image Captioningcited by 79
  9. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  10. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  11. 2019Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignmentcited by 51
  12. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  13. 2020Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Groundingcited by 50
  14. 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
  15. 2020DeVLBert: Learning Deconfounded Visio-Linguistic Representationscited by 42
  16. 2017Weakly-supervised Visual Grounding of Phrases with Linguistic Structurescited by 40
  17. 2023In-Context Learning with Many Demonstration Examplescited by 39
  18. 2019Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 37
  19. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  20. 2018Knowledge Aided Consistency for Weakly Supervised Phrase Groundingcited by 32
  21. 2019Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 21
  22. 2020Iterative Context-Aware Graph Inference for Visual Dialogcited by 16
  23. 2023GPT Self-Supervision for a Better Data Annotatorcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning
  2. 2025Causal Prompting for Implicit Sentiment Analysis with Large Language Models
  3. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  4. 2026Improving Implicit Discourse Relation Recognition with Natural Language Explanations from LLMs
  5. 2025Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition
  6. 2024Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities
  7. 2022Reducing the Vision and Language Bias for Temporal Sentence Grounding
  8. 2025Cross-modal Causal Relation Alignment for Video Question Grounding