# Question-controlled Text-aware Image Captioning arXiv: https://arxiv.org/abs/2108.02059 ยท HTML: https://mlsanity.com/paper/2108.02059/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 35 references that are in the map. - First posted on arXiv 4 Aug 2021. ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014) - [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016) - [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016) - [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020) - [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019) - [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019) - [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016) - [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019) - [Captioning Images Taken by People Who Are Blind](https://mlsanity.com/paper/2002.08565/) (2020) - [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019) - [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020) - [Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs](https://mlsanity.com/paper/2003.00387/) (2020) - [Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions](https://mlsanity.com/paper/1811.10652/) (2018) - [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018) - [Describing like humans: on diversity in image captioning](https://mlsanity.com/paper/1903.12020/) (2019) - [Spatially Aware Multimodal Transformers for TextVQA](https://mlsanity.com/paper/2007.12146/) (2020) - [Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning](https://mlsanity.com/paper/1908.08529/) (2019) - [Intention Oriented Image Captions with Guiding Objects](https://mlsanity.com/paper/1811.07662/) (2018) - [Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps](https://mlsanity.com/paper/2012.05153/) (2020) - [Confidence-aware Non-repetitive Multimodal Transformers for TextCaps](https://mlsanity.com/paper/2012.03662/) (2020) ## Built on by (most cited first, 9 of 9) - [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024) - [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023) - [UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model](https://mlsanity.com/paper/2310.05126/) (2023) - [mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding](https://mlsanity.com/paper/2409.03420/) (2024) - [InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation](https://mlsanity.com/paper/2305.06002/) (2023) - [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023) - [ChiQA: A Large Scale Image-based Real-World Question Answering Dataset for Multi-Modal Understanding](https://mlsanity.com/paper/2208.03030/) (2022) - [Explore and Tell: Embodied Visual Captioning in 3D Environments](https://mlsanity.com/paper/2308.10447/) (2023) - [MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysis](https://mlsanity.com/paper/2407.02842/) (2024) ## Abstract (arXiv) For an image with multiple scene texts, different people may be interested in different text information. Current text-aware image captioning models are not able to generate distinctive captions according to various information needs. To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap). With questions as control signals, this task requires models to understand questions, find related scene texts and describe them together with objects fluently in human language. Based on two existing text-aware captioning datasets, we automatically construct two datasets, ControlTextCaps and ControlVizWiz to support the task. We propose a novel Geometry and Question Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to fuse region-level object features and region-level scene text features with considering spatial relationships. Then, we design a Question-guided Encoder to select the most relevant visual features for each question. Finally, GQAM generates a personalized text-aware caption with a Multimodal Decoder. Our model achieves better captioning performance and question answering ability than carefully designed baselines on both two datasets. With questions as control signals, our model generates more informative and diverse captions than the state-of-the-art text-aware captioning model. Our code and datasets are publicly available at https://github.com/HAWLYQ/Qc-TextCap.