# VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning

arXiv: https://arxiv.org/abs/2303.02635 · HTML: https://mlsanity.com/paper/2303.02635/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 5 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [Exploring Models and Data for Image Question Answering](https://mlsanity.com/paper/1505.02074/) (2015)
- [A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input](https://mlsanity.com/paper/1410.0210/) (2014)
- [Explicit Knowledge-based Reasoning for Visual Question Answering](https://mlsanity.com/paper/1511.02570/) (2015)
- [Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering](https://mlsanity.com/paper/1505.05612/) (2015)
- [MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding](https://mlsanity.com/paper/2112.10728/) (2021)

## Built on by (most cited first, 7 of 7)

- [ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior](https://mlsanity.com/paper/2512.05745/) (2025)
- [Investigating Mechanisms for In-Context Vision Language Binding](https://mlsanity.com/paper/2505.22200/) (2025)
- [ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions](https://mlsanity.com/paper/2509.05066/) (2025)
- [From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning](https://mlsanity.com/paper/2502.05843/) (2025)
- [Attention Grounded Enhancement for Visual Document Retrieval](https://mlsanity.com/paper/2511.13415/) (2025)
- [RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation](https://mlsanity.com/paper/2603.14880/) (2026)
- [From Transformation to Target State: Rethinking Query Representation for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2610.05993/) (2026)

## Abstract (arXiv)

The ideal form of Visual Question Answering requires understanding, grounding and reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most existing VQA benchmarks are limited to just picking the answer from a pre-defined set of options and lack attention to text. We present a new challenge with a dataset that contains 23,781 questions based on 10124 image-text pairs. Specifically, the task requires the model to align multimedia representations of the same entity to implement multi-hop reasoning between image and text and finally use natural language to answer the question. The aim of this challenge is to develop and benchmark models that are capable of multimedia entity alignment, multi-step reasoning and open-ended answer generation.