# Modularized Textual Grounding for Counterfactual Resilience

arXiv: https://arxiv.org/abs/1904.03589 · HTML: https://mlsanity.com/paper/1904.03589/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2019.

## Built on by (most cited first, 17 of 17)

- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images](https://mlsanity.com/paper/2103.07894/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Deconfounded Visual Grounding](https://mlsanity.com/paper/2112.15324/) (2021)
- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [Contrastive Learning with Counterfactual Explanations for Radiology Report Generation](https://mlsanity.com/paper/2407.14474/) (2024)
- [VCD: Visual Causality Discovery for Cross-Modal Question Reasoning](https://mlsanity.com/paper/2304.08083/) (2023)
- [Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement](https://mlsanity.com/paper/2305.10913/) (2023)
- [From Coarse to Fine-grained Concept based Discrimination for Phrase Detection](https://mlsanity.com/paper/2112.03237/) (2021)
- [Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos](https://mlsanity.com/paper/2204.13548/) (2022)
- [Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations](https://mlsanity.com/paper/2605.09090/) (2026)

## Abstract (arXiv)

Computer Vision applications often require a textual grounding module with precision, interpretability, and resilience to counterfactual inputs/queries. To achieve high grounding precision, current textual grounding methods heavily rely on large-scale training data with manual annotations at the pixel level. Such annotations are expensive to obtain and thus severely narrow the model's scope of real-world applications. Moreover, most of these methods sacrifice interpretability, generalizability, and they neglect the importance of being resilient to counterfactual inputs. To address these issues, we propose a visual grounding system which is 1) end-to-end trainable in a weakly supervised fashion with only image-level annotations, and 2) counterfactually resilient owing to the modular design. Specifically, we decompose textual descriptions into three levels: entity, semantic attribute, color information, and perform compositional grounding progressively. We validate our model through a series of experiments and demonstrate its improvement over the state-of-the-art methods. In particular, our model's performance not only surpasses other weakly/un-supervised methods and even approaches the strongly supervised ones, but also is interpretable for decision making and performs much better in face of counterfactual classes than all the others.