mlsanityData to 14 Oct 2026

Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations

arXiv 2306.08658 ↗First posted 14 Jun 2023cs.CL cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision-and-language (VL) models with separate encoders for each modality (e.g., CLIP) have become the go-to models for zero-shot image classification and image-text retrieval. They are, however, mostly evaluated in English as multilingual benchmarks are limited in availability. We introduce Babel-ImageNet, a massively multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation. We instead automatically obtain reliable translations by linking them -- via shared WordNet synsets -- to BabelNet, a massively multilingual lexico-semantic network. We evaluate 11 public multilingual CLIP models on zero-shot image classification (ZS-IC) on our benchmark, demonstrating a significant gap between English ImageNet performance and that of high-resource languages (e.g., German or Chinese), and an even bigger gap for low-resource languages (e.g., Sinhala or Lao). Crucially, we show that the models' ZS-IC performance highly correlates with their performance in image-text retrieval, validating the use of Babel-ImageNet to evaluate multilingual models for the vast majority of languages without gold image-text data. Finally, we show that the performance of multilingual CLIP can be drastically improved for low-resource languages with parameter-efficient language-specific training. We make our code and data publicly available: urlhttps://github.com/gregor-ge/Babel-ImageNet

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

52 of 71 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  12. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  13. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  14. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  15. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  16. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  17. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  18. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  19. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  20. 2019Does Object Recognition Work for Everyone?cited by 151
  21. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  22. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  23. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  24. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  25. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  26. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  27. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  28. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  29. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  30. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  31. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  32. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 28
  33. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  34. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  35. 2019MULE: Multimodal Universal Language Embeddingcited by 22
  36. 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22
  37. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  38. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 4; 2026: 3.

11 papers.

  1. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  3. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  4. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  5. 2024Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Costcited by 5
  6. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  7. 2025xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillationcited by 3
  8. 2026DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Modelscited by 3
  9. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Modelscited by 2
  10. 2026IchthyoNoma: Nomenclature and Context Sensitivity of Zero-Shot Biological Vision--Language Models for Bangladeshi Freshwater Fish Recognitioncited by 0
  11. 2026SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languagescited by 0