# Multi-Modal Fusion Transformer for Visual Question Answering in Remote Sensing

arXiv: https://arxiv.org/abs/2210.04510 · HTML: https://mlsanity.com/paper/2210.04510/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [xView: Objects in Context in Overhead Imagery](https://mlsanity.com/paper/1802.07856/) (2018)
- [How to find a good image-text embedding for remote sensing visual question answering?](https://mlsanity.com/paper/2109.11848/) (2021)

## Built on by (most cited first, 13 of 13)

- [MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition](https://mlsanity.com/paper/2305.07214/) (2023)
- [HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language](https://mlsanity.com/paper/2305.17690/) (2023)
- [RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2310.13120/) (2023)
- [LiT-4-RSVQA: Lightweight Transformer-based Visual Question Answering in Remote Sensing](https://mlsanity.com/paper/2306.00758/) (2023)
- [Segmentation-guided Attention for Visual Question Answering from Remote Sensing Images](https://mlsanity.com/paper/2407.08669/) (2024)
- [GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing](https://mlsanity.com/paper/2601.17089/) (2026)
- [Text-Guided Coarse-to-Fine Fusion Network for Robust Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2411.15770/) (2024)
- [The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation](https://mlsanity.com/paper/2311.16782/) (2023)
- [From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing](https://mlsanity.com/paper/2411.05826/) (2024)
- [SAR Strikes Back: A New Hope for RSVQA](https://mlsanity.com/paper/2501.08131/) (2025)
- [Checkmate: interpretable and explainable RSVQA is the endgame](https://mlsanity.com/paper/2508.13086/) (2025)
- [Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities](https://mlsanity.com/paper/2504.08578/) (2025)
- [Copy-Move Forgery Detection and Question Answering for Remote Sensing Image](https://mlsanity.com/paper/2412.02575/) (2024)

## Abstract (arXiv)

With the new generation of satellite technologies, the archives of remote sensing (RS) images are growing very fast. To make the intrinsic information of each RS image easily accessible, visual question answering (VQA) has been introduced in RS. VQA allows a user to formulate a free-form question concerning the content of RS images to extract generic information. It has been shown that the fusion of the input modalities (i.e., image and text) is crucial for the performance of VQA systems. Most of the current fusion approaches use modality-specific representations in their fusion modules instead of joint representation learning. However, to discover the underlying relation between both the image and question modality, the model is required to learn the joint representation instead of simply combining (e.g., concatenating, adding, or multiplying) the modality-specific representations. We propose a multi-modal transformer-based architecture to overcome this issue. Our proposed architecture consists of three main modules: i) the feature extraction module for extracting the modality-specific features; ii) the fusion module, which leverages a user-defined number of multi-modal transformer layers of the VisualBERT model (VB); and iii) the classification module to obtain the answer. Experimental results obtained on the RSVQAxBEN and RSVQA-LR datasets (which are made up of RGB bands of Sentinel-2 images) demonstrate the effectiveness of VBFusion for VQA tasks in RS. To analyze the importance of using other spectral bands for the description of the complex content of RS images in the framework of VQA, we extend the RSVQAxBEN dataset to include all the spectral bands of Sentinel-2 images with 10m and 20m spatial resolution.