mlsanityData to 7 Oct 2026

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

arXiv 2201.11732 ↗First posted 27 Jan 2022cs.CL cs.CVcited by 54

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of a multilingual benchmark, however, vision-and-language research has mostly focused on English language tasks. To fill this gap, we introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together - by both aggregating pre-existing datasets and creating new ones - visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups. Based on the evaluation of the available state-of-the-art models, we find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks. Moreover, downstream performance is partially explained by the amount of available unlabelled textual data for pretraining, and only weakly by the typological distance of target-source languages. We hope to encourage future research efforts in this area by releasing the benchmark to the community.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 75 of its 108 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

75 of 108 references.

References · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  14. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  15. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  16. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  18. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  19. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  20. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  21. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  22. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  23. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  24. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  25. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  26. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  27. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  28. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  29. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  30. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  31. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  32. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  33. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  34. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  35. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  36. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  37. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  38. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 12; 2024: 11; 2025: 11; 2026: 4.

54 papers.

Sort:

By area:

  1. 2022Visual Spatial Reasoningcited by 288
  2. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  3. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  4. 2023Modular Deep Learningcited by 82
  5. 2022Language Modelling with Pixelscited by 53
  6. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  7. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  8. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  9. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  10. 2022Combining Modular Skills in Multitask Learningcited by 27
  11. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  12. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
38 more
  1. 2022ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culturecited by 17
  2. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  3. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  4. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  5. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  6. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  7. 2024DevBench: A multimodal developmental benchmark for language learningcited by 13
  8. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  9. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methodscited by 11
  10. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  11. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  12. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  13. 2022MaXM: Towards Multilingual Visual Question Answeringcited by 9
  14. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  15. 2023Translation-Enhanced Multilingual Text-to-Image Generationcited by 9
  16. 2022Delving Deeper into Cross-lingual Visual Question Answeringcited by 8
  17. 2024mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpuscited by 8
  18. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generationcited by 7
  19. 2024Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learningcited by 7
  20. 2024BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrievalcited by 6
  21. 2022Improving the Cross-Lingual Generalisation in Visual Question Answeringcited by 5
  22. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  23. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5
  24. 2023bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmarkcited by 4
  25. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  26. 2023Meta-learning For Vision-and-language Cross-lingual Transfercited by 3
  27. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  28. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  29. 2023RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-trainingcited by 2
  30. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languagescited by 2
  31. 2025Topological Alignment of Shared Vision-Language Embedding Spacecited by 2
  32. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1
  33. 2024Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learningcited by 1
  34. 2025RusCode: Russian Cultural Code Benchmark for Text-to-Image Generationcited by 1
  35. 2025Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoderscited by 1
  36. 2025Multilingual Vision-Language Models, A Surveycited by 1
  37. 2025POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarioscited by 1
  38. 2024Green Runner: A tool for efficient deep learning component selectioncited by 0

50 most cited of 54 All 54 (JSON)