mlsanityData to 14 Oct 2026

AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

arXiv 2211.06679 ↗First posted 12 Nov 2022cs.CLcited by 85

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model. Starting from the pre-trained multimodal representation model CLIP released by OpenAI, we altered its text encoder with a pre-trained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k-CN, COCO-CN and XTD. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 43 of its 77 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

43 of 77 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2017The Kinetics Human Action Video Datasetcited by 1,909
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
31 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  8. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  12. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  13. 2018A Short Note about Kinetics-600cited by 333
  14. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  15. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  16. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  17. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  18. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  19. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  20. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  21. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  22. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  23. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  24. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  25. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  26. 2017Moonshine: Distilling with Cheap Convolutionscited by 25
  27. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  28. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  29. 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12
  30. 2019Image search using multilingual texts: a cross-modal learning approach between image and textcited by 7
  31. 2022Generalizing Multimodal Pre-training into Multilingual via Language Acquisitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 25; 2024: 28; 2025: 19; 2026: 12.

85 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 151
  4. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Modelscited by 66
  5. 2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 63
  6. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 62
  7. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  8. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  9. 2022On the Power of Foundation Modelscited by 37
  10. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 32
  11. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Modelcited by 30
  12. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
38 more
  1. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  2. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  3. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  4. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  5. 2023Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?cited by 16
  6. 2023Foundation Model is Efficient Multimodal Multitask Model Selectorcited by 16
  7. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  8. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 15
  9. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  10. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  11. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  12. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  13. 2024Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Supportcited by 11
  14. 2023On the Cultural Gap in Text-to-Image Generationcited by 10
  15. 2023Multilingual Conceptual Coverage in Text-to-Image Modelscited by 9
  16. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  17. 2024Clarify: Improving Model Robustness With Natural Language Correctionscited by 8
  18. 2025NTIRE 2025 challenge on Text to Image Generation Model Quality Assessmentcited by 8
  19. 2023CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languagescited by 7
  20. 2024PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answeringcited by 7
  21. 2024ColorFoil: Investigating Color Blindness in Large Vision and Language Modelscited by 7
  22. 2023Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communitiescited by 6
  23. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levelscited by 5
  24. 2023PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloudcited by 5
  25. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generationcited by 5
  26. 2024M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrievalcited by 5
  27. 2024Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Conceptscited by 5
  28. 2024Lightweight Unsupervised Federated Learning with Pretrained Vision Language Modelcited by 5
  29. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  30. 2024Navigating Text-to-Image Generative Bias across Indic Languagescited by 5
  31. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Modelscited by 5
  32. 2025Enhanced Cross-modal 3D Retrieval via Tri-modal Reconstructioncited by 5
  33. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selectioncited by 5
  34. 2026RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Detailscited by 5
  35. 2023Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivotscited by 4
  36. 2023iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Designcited by 4
  37. 2024M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretrainingcited by 4
  38. 2024What If the Input is Expanded in OOD Detection?cited by 4

50 most cited of 85 All 85 (JSON)