# Fusion of Detected Objects in Text for Visual Question Answering

arXiv: https://arxiv.org/abs/1908.05054 · HTML: https://mlsanity.com/paper/1908.05054/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 30 of 74)

- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks](https://mlsanity.com/paper/2112.01522/) (2021)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [Kaleido-BERT: Vision-Language Pre-training on Fashion Domain](https://mlsanity.com/paper/2103.16110/) (2021)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Contrastive Language-Image Pre-Training with Knowledge Graphs](https://mlsanity.com/paper/2210.08901/) (2022)
- [Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information](https://mlsanity.com/paper/2211.09807/) (2022)
- [Learning to Answer Visual Questions from Web Videos](https://mlsanity.com/paper/2205.05019/) (2022)
- [PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models](https://mlsanity.com/paper/2205.11169/) (2022)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch](https://mlsanity.com/paper/2208.03354/) (2022)
- [PreSTU: Pre-Training for Scene-Text Understanding](https://mlsanity.com/paper/2209.05534/) (2022)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration](https://mlsanity.com/paper/2203.04006/) (2022)
- [UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models](https://mlsanity.com/paper/2310.10942/) (2023)

## Abstract (arXiv)

To advance models of multimodal context, we introduce a simple yet powerful neural architecture for data that combines vision and natural language. The "Bounding Boxes in Text Transformer" (B2T2) also leverages referential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Commonsense Reasoning benchmark (https://visualcommonsense.com), achieving a new state-of-the-art with a 25% relative reduction in error rate compared to published baselines and obtaining the best performance to date on the public leaderboard (as of May 22, 2019). A detailed ablation analysis shows that the early integration of the visual features into the text analysis is key to the effectiveness of the new architecture. A reference implementation of our models is provided (https://github.com/google-research/language/tree/master/language/question_answering/b2t2).