# Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA

arXiv: https://arxiv.org/abs/2310.09147 · HTML: https://mlsanity.com/paper/2310.09147/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships](https://mlsanity.com/paper/2003.00504/) (2020)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering](https://mlsanity.com/paper/2002.10215/) (2020)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [Pythia v0.1: the Winning Entry to the VQA Challenge 2018](https://mlsanity.com/paper/1807.09956/) (2018)
- [Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering](https://mlsanity.com/paper/2006.09073/) (2020)
- [ICDAR 2019 Competition on Scene Text Visual Question Answering](https://mlsanity.com/paper/1907.00490/) (2019)
- [Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text](https://mlsanity.com/paper/2003.13962/) (2020)
- [Spatially Aware Multimodal Transformers for TextVQA](https://mlsanity.com/paper/2007.12146/) (2020)
- [Omnidirectional Scene Text Detection with Sequential-free Box Discretization](https://mlsanity.com/paper/1906.02371/) (2019)
- [Relationship-Embedded Representation Learning for Grounding Referring Expressions](https://mlsanity.com/paper/1906.04464/) (2019)

## Built on by (most cited first, 6 of 6)

- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting](https://mlsanity.com/paper/2402.15070/) (2024)
- [Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination](https://mlsanity.com/paper/2511.17490/) (2025)
- [ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese](https://mlsanity.com/paper/2310.18046/) (2023)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [InstructHumans: Editing Animated 3D Human Textures with Instructions](https://mlsanity.com/paper/2404.04037/) (2024)

## Abstract (arXiv)

Text-based visual question answering (TextVQA) faces the significant challenge of avoiding redundant relational inference. To be specific, a large number of detected objects and optical character recognition (OCR) tokens result in rich visual relationships. Existing works take all visual relationships into account for answer prediction. However, there are three observations: (1) a single subject in the images can be easily detected as multiple objects with distinct bounding boxes (considered repetitive objects). The associations between these repetitive objects are superfluous for answer reasoning; (2) two spatially distant OCR tokens detected in the image frequently have weak semantic dependencies for answer reasoning; and (3) the co-existence of nearby objects and tokens may be indicative of important visual cues for predicting answers. Rather than utilizing all of them for answer prediction, we make an effort to identify the most important connections or eliminate redundant ones. We propose a sparse spatial graph network (SSGN) that introduces a spatially aware relation pruning technique to this task. As spatial factors for relation measurement, we employ spatial distance, geometric dimension, overlap area, and DIoU for spatially aware pruning. We consider three visual relationships for graph learning: object-object, OCR-OCR tokens, and object-OCR token relationships. SSGN is a progressive graph learning architecture that verifies the pivotal relations in the correlated object-token sparse graph, and then in the respective object-based sparse graph and token-based sparse graph. Experiment results on TextVQA and ST-VQA datasets demonstrate that SSGN achieves promising performances. And some visualization results further demonstrate the interpretability of our method.