mlsanityData to 14 Oct 2026

On the Difference of BERT-style and CLIP-style Text Encoders

arXiv 2306.03678 ↗First posted 6 Jun 2023cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks. However, few studies are dedicated to studying the text encoders learned by CLIP. In this paper, we analyze the difference between BERT-style and CLIP-style text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation. Experimental analyses show that although CLIP-style text encoders underperform BERT-style ones for general text understanding tasks, they are equipped with a unique ability, i.e., synesthesia, for the cross-modal association, which is more similar to the senses of humans.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

22 of 37 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
10 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  10. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 5.

15 papers.

Sort:

By area:

  1. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  2. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  3. 2023Reverse Stable Diffusion: What prompt was used to generate this image?cited by 7
  4. 2024Leveraging Foundation Models for Zero-Shot IoT Sensingcited by 4
  5. 2023Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification with Cross-Modal Retrievalcited by 3
  6. 2025On the Brittleness of CLIP Text Encoderscited by 3
  7. 2024Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-taskscited by 1
  8. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
  9. 2024LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classificationcited by 1
  10. 2024Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairnesscited by 1
  11. 2024Unified Framework for Open-World Compositional Zero-shot Learningcited by 1
  12. 2025Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inferencecited by 1
3 more
  1. 2025Decoder-Only LLMs are Better Controllers for Diffusion Modelscited by 0
  2. 2025Visual moral inference and communicationcited by 0
  3. 2025Virtual Dosimetrists: A Radiotherapy Training "Flight Simulator"cited by 0