# Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering

arXiv: https://arxiv.org/abs/1909.02097 · HTML: https://mlsanity.com/paper/1909.02097/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering](https://mlsanity.com/paper/2205.00949/) (2022)
- [Object-Centric Unsupervised Image Captioning](https://mlsanity.com/paper/2112.00969/) (2021)
- [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)

## Abstract (arXiv)

Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering. However, popular models like Faster R-CNN rely on a costly process of annotating ground-truths for both the bounding boxes and their corresponding semantic labels, making it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and featurization for down-stream tasks. The key insight is that this allows us to leverage a large amount of labeled annotations that were previously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and improved image captioning and visual question answering models, as measured on publicly available benchmarks.