# xGQA: Cross-Lingual Visual Question Answering

arXiv: https://arxiv.org/abs/2109.06082 · HTML: https://mlsanity.com/paper/2109.06082/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 86 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)

## Built on by (most cited first, 30 of 72)

- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering](https://mlsanity.com/paper/2405.11985/) (2024)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Meta CLIP 2: A Worldwide Scaling Recipe](https://mlsanity.com/paper/2507.22062/) (2025)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://mlsanity.com/paper/2308.12038/) (2023)
- [mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs](https://mlsanity.com/paper/2307.06930/) (2023)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture](https://mlsanity.com/paper/2406.11030/) (2024)
- [Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training](https://mlsanity.com/paper/2206.00621/) (2022)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation](https://mlsanity.com/paper/2410.17250/) (2024)
- [Multilingual Multimodal Learning with Machine Translated Text](https://mlsanity.com/paper/2210.13134/) (2022)
- [No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models](https://mlsanity.com/paper/2405.13777/) (2024)
- [Multilingual Diversity Improves Vision-Language Representations](https://mlsanity.com/paper/2405.16915/) (2024)
- [HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language](https://mlsanity.com/paper/2305.17690/) (2023)
- [Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations](https://mlsanity.com/paper/2306.08658/) (2023)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [Conditional computation in neural networks: principles and research trends](https://mlsanity.com/paper/2403.07965/) (2024)
- [The Multilingual Divide and Its Impact on Global AI Safety](https://mlsanity.com/paper/2505.21344/) (2025)
- [MaXM: Towards Multilingual Visual Question Answering](https://mlsanity.com/paper/2209.05401/) (2022)
- [WildQA: In-the-Wild Video Question Answering](https://mlsanity.com/paper/2209.06650/) (2022)
- [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024)
- [Delving Deeper into Cross-lingual Visual Question Answering](https://mlsanity.com/paper/2202.07630/) (2022)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)

## Abstract (arXiv)

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and -- vice versa -- multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling.