mlsanityData to 7 Oct 2026

Visually Grounded Reasoning across Languages and Cultures

arXiv 2109.13238 ↗First posted 28 Sep 2021cs.CL cs.AI cs.CVcited by 188

Not on the map. Why?

Cited by 65 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet. While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias. Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. In particular, we let the selection of both concepts and images be entirely driven by native speakers, rather than scraping them automatically. Specifically, we focus on a typologically diverse set of languages, namely, Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish. On top of the concepts and images obtained through this new protocol, we create a multilingual dataset for {M}ulticultur{a}l {R}easoning over {V}ision and {L}anguage (MaRVL) by eliciting statements from native speaker annotators about pairs of images. The task consists of discriminating whether each grounded statement is true or false. We establish a series of baselines using state-of-the-art models and find that their cross-lingual transfer performance lags dramatically behind supervised performance in English. These results invite us to reassess the robustness and accuracy of current state-of-the-art models beyond a narrow domain, but also open up new exciting challenges for the development of truly multilingual and multicultural systems.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Builds on 45 of its 85 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

45 of 85 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
33 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  9. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  10. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  11. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  13. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  14. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  15. 2016What makes ImageNet good for transfer learning?cited by 192
  16. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  17. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  18. 2019Does Object Recognition Work for Everyone?cited by 149
  19. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  20. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  21. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  22. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  23. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  24. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  25. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  26. 2020Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similaritycited by 38
  27. 2015Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learningcited by 18
  28. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16
  29. 2020Internal and external pressures on language emergence: least effort, object constancy and frequencycited by 13
  30. 2020Emergent Communication Pretraining for Few-Shot Machine Translationcited by 13
  31. 2017Cross-linguistic differences and similarities in image descriptionscited by 12
  32. 2021Minimax and Neyman-Pearson Meta-Learning for Outlier Languagescited by 10
  33. 2018Linking ImageNet WordNet Synsets with Wikidatacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 35; 2023: 39; 2024: 48; 2025: 43; 2026: 22.

188 papers.

Sort:

By area:

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2022Visual Spatial Reasoningcited by 288
  3. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  4. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  7. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  8. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  9. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  10. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  11. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  12. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
38 more
  1. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  2. 2023Modular Deep Learningcited by 82
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  4. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  5. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  6. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  7. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  8. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  9. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  10. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  11. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  12. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  13. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  14. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  15. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  16. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  17. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  18. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  19. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
  20. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  21. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  22. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  23. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 31
  24. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  25. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  26. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  27. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  28. 2022Performance Optimization for Semantic Communications: An Attention-based Reinforcement Learning Approachcited by 28
  29. 2022What a Creole Wants, What a Creole Needscited by 27
  30. 2023StoryBench: A Multifaceted Benchmark for Continuous Story Visualizationcited by 27
  31. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  32. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  33. 2022Image Retrieval from Contextual Descriptionscited by 26
  34. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  35. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  36. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  37. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  38. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24

50 most cited of 188 All 188 (JSON)