# Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps

arXiv: https://arxiv.org/abs/2012.05153 · HTML: https://mlsanity.com/paper/2012.05153/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval](https://mlsanity.com/paper/2203.16778/) (2022)
- [Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering](https://mlsanity.com/paper/2209.10326/) (2022)
- [ICDAR 2021 Competition on Document VisualQuestion Answering](https://mlsanity.com/paper/2111.05547/) (2021)
- [Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling](https://mlsanity.com/paper/2108.08965/) (2021)
- [TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation](https://mlsanity.com/paper/2208.01813/) (2022)
- [Question-controlled Text-aware Image Captioning](https://mlsanity.com/paper/2108.02059/) (2021)
- [Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA](https://mlsanity.com/paper/2310.09147/) (2023)
- [MUST-VQA: MUltilingual Scene-text VQA](https://mlsanity.com/paper/2209.06730/) (2022)
- [Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA](https://mlsanity.com/paper/2304.01603/) (2023)
- [One for All: One-stage Referring Expression Comprehension with Dynamic Reasoning](https://mlsanity.com/paper/2208.00361/) (2022)
- [DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based Image Captioning](https://mlsanity.com/paper/2302.01540/) (2023)
- [Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering](https://mlsanity.com/paper/2203.12929/) (2022)
- [SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering](https://mlsanity.com/paper/2212.08283/) (2022)
- [Image-Text Pre-Training for Logo Recognition](https://mlsanity.com/paper/2309.10206/) (2023)
- [Towards Multimodal Vision-Language Models Generating Non-Generic Text](https://mlsanity.com/paper/2207.04174/) (2022)
- [Text-Aware Dual Routing Network for Visual Question Answering](https://mlsanity.com/paper/2211.14450/) (2022)

## Abstract (arXiv)

Texts appearing in daily scenes that can be recognized by OCR (Optical Character Recognition) tools contain significant information, such as street name, product brand and prices. Two tasks -- text-based visual question answering and text-based image captioning, with a text extension from existing vision-language applications, are catching on rapidly. To address these problems, many sophisticated multi-modality encoding frameworks (such as heterogeneous graph structure) are being used. In this paper, we argue that a simple attention mechanism can do the same or even better job without any bells and whistles. Under this mechanism, we simply split OCR token features into separate visual- and linguistic-attention branches, and send them to a popular Transformer decoder to generate answers or captions. Surprisingly, we find this simple baseline model is rather strong -- it consistently outperforms state-of-the-art (SOTA) models on two popular benchmarks, TextVQA and all three tasks of ST-VQA, although these SOTA models use far more complex encoding mechanisms. Transferring it to text-based image captioning, we also surpass the TextCaps Challenge 2020 winner. We wish this work to set the new baseline for this two OCR text related applications and to inspire new thinking of multi-modality encoder design. Code is available at https://github.com/ZephyrZhuQi/ssbaseline