# Separate and Locate: Rethink the Text in Text-based Visual Question Answering

arXiv: https://arxiv.org/abs/2308.16383 · HTML: https://mlsanity.com/paper/2308.16383/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering](https://mlsanity.com/paper/2002.10215/) (2020)
- [Fusion of Detected Objects in Text for Visual Question Answering](https://mlsanity.com/paper/1908.05054/) (2019)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [Spatially Aware Multimodal Transformers for TextVQA](https://mlsanity.com/paper/2007.12146/) (2020)
- [PreSTU: Pre-Training for Scene-Text Understanding](https://mlsanity.com/paper/2209.05534/) (2022)
- [Structured Multimodal Attentions for TextVQA](https://mlsanity.com/paper/2006.00753/) (2020)
- [Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering](https://mlsanity.com/paper/2010.02582/) (2020)
- [Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling](https://mlsanity.com/paper/2108.08965/) (2021)
- [TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation](https://mlsanity.com/paper/2208.01813/) (2022)
- [Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering](https://mlsanity.com/paper/2203.12929/) (2022)

## Built on by (most cited first, 7 of 7)

- [ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images](https://mlsanity.com/paper/2404.18397/) (2024)
- [ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Images](https://mlsanity.com/paper/2404.10652/) (2024)
- [GUIDE: Graphical User Interface Data for Execution](https://mlsanity.com/paper/2404.16048/) (2024)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receipts](https://mlsanity.com/paper/2502.19202/) (2025)
- [Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark](https://mlsanity.com/paper/2512.22218/) (2025)
- [LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning](https://mlsanity.com/paper/2604.02338/) (2026)

## Abstract (arXiv)

Text-based Visual Question Answering (TextVQA) aims at answering questions about the text in images. Most works in this field focus on designing network structures or pre-training tasks. All these methods list the OCR texts in reading order (from left to right and top to bottom) to form a sequence, which is treated as a natural language ``sentence''. However, they ignore the fact that most OCR words in the TextVQA task do not have a semantical contextual relationship. In addition, these approaches use 1-D position embedding to construct the spatial relation between OCR tokens sequentially, which is not reasonable. The 1-D position embedding can only represent the left-right sequence relationship between words in a sentence, but not the complex spatial position relationship. To tackle these problems, we propose a novel method named Separate and Locate (SaL) that explores text contextual cues and designs spatial position embedding to construct spatial relations between OCR texts. Specifically, we propose a Text Semantic Separate (TSS) module that helps the model recognize whether words have semantic contextual relations. Then, we introduce a Spatial Circle Position (SCP) module that helps the model better construct and reason the spatial position relationships between OCR texts. Our SaL model outperforms the baseline model by 4.44% and 3.96% accuracy on TextVQA and ST-VQA datasets. Compared with the pre-training state-of-the-art method pre-trained on 64 million pre-training samples, our method, without any pre-training tasks, still achieves 2.68% and 2.52% accuracy improvement on TextVQA and ST-VQA. Our code and models will be released at https://github.com/fangbufang/SaL.