# Cross-Modality Relevance for Reasoning on Language and Vision

arXiv: https://arxiv.org/abs/2005.06035 · HTML: https://mlsanity.com/paper/2005.06035/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 10 of 10)

- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Panoptic Narrative Grounding](https://mlsanity.com/paper/2109.04988/) (2021)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [Learning to Name Classes for Vision and Language Models](https://mlsanity.com/paper/2304.01830/) (2023)
- [Relational Gating for "What If" Reasoning](https://mlsanity.com/paper/2105.13449/) (2021)
- [Relevant CommonSense Subgraphs for "What if..." Procedural Reasoning](https://mlsanity.com/paper/2203.11187/) (2022)
- [Labeling Comic Mischief Content in Online Videos with a Multimodal Hierarchical-Cross-Attention Model](https://mlsanity.com/paper/2406.07841/) (2024)
- [Towards Automated Petrography](https://mlsanity.com/paper/2511.00328/) (2025)

## Abstract (arXiv)

This work deals with the challenge of learning and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR). We design a novel cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. The learned alignments of input spaces and their relevance representations by NLVR task boost the training efficiency of VQA task.