mlsanityData to 7 Oct 2026

Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning

arXiv 2505.13628 ↗First posted 19 May 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very easy to create without requiring multilingual expertise, so this offers a more efficient alternative for low-resource languages. We find that multilingual image-caption alignment can implicitly align the text representations between languages, languages unseen by the encoder in pretraining can be incorporated into this alignment post-hoc, and these aligned representations are usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

23 of 33 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  10. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
11 more
  1. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  2. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  3. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  4. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  5. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  6. 2024What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialectscited by 20
  7. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  8. 2018Unsupervised Multilingual Word Embeddingscited by 16
  9. 2021Bilingual alignment transfers to multilingual alignment for unsupervised parallel text miningcited by 4
  10. 2020Practical Comparable Data Collection for Low-Resource Languages via Imagescited by 3
  11. 2018Towards Language Agnostic Universal Representationscited by 1

Built on by

Not cited yet.