mlsanityData to 7 Oct 2026

CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languages

arXiv 2310.13683 ↗First posted 20 Oct 2023cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This work introduces CAPIVARA, a cost-efficient framework designed to enhance the performance of multilingual CLIP models in low-resource languages. While CLIP has excelled in zero-shot vision-language tasks, the resource-intensive nature of model training remains challenging. Many datasets lack linguistic diversity, featuring solely English descriptions for images. CAPIVARA addresses this by augmenting text data using image captioning and machine translation to generate multiple synthetic captions in low-resource languages. We optimize the training pipeline with LiT, LoRA, and gradient checkpointing to alleviate the computational cost. Through extensive experiments, CAPIVARA emerges as state of the art in zero-shot tasks involving images and Portuguese texts. We show the potential for significant improvements in other low-resource languages, achieved by fine-tuning the pre-trained multilingual CLIP using CAPIVARA on a single GPU for 2 hours. Our model and code is available at https://github.com/hiaac-nlp/CAPIVARA.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 35 of its 60 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

35 of 60 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2018Model Cards for Model Reportingcited by 1,068
23 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  6. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  7. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  8. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  9. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  10. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  11. 2023Improving CLIP Training with Language Rewritescited by 216
  12. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  13. 2018Marian: Fast Neural Machine Translation in C++cited by 202
  14. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  15. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  16. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  17. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  18. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  19. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  20. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  21. 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12
  22. 2021EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modelingcited by 10
  23. 2021#PraCegoVer: A Large Dataset for Image Captioning in Portuguesecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  2. 2025Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountabilitycited by 6
  3. 2024FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Modelscited by 4
  4. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  5. 2024CLIP model is an Efficient Online Lifelong Learnercited by 2
  6. 2025Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Studycited by 0
  7. 2026Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Datasetcited by 0