mlsanityData to 14 Oct 2026

TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge

arXiv 2510.21879 ↗First posted 23 Oct 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent years have witnessed an increasing interest in image-text contrastive modeling, exemplified by models such as Contrastive Language-Image Pretraining (CLIP). In this paper, we propose the TernaryCLIP, a lightweight computational framework that converts connection weights of both vision and text encoders of CLIP into the ternary format, instead of full-precision or floating ones. TernaryCLIP incorporates quantization-aware training and distillation modules, preventing precision degradation and enabling low-cost and high-efficiency computations. Comprehensive experiments demonstrate that TernaryCLIP can achieve up to 99% ternarized weights with 1.58-bit representation, 16.98 × compression ratio, 2.3 × inference acceleration, 16 × storage reduction, 10 × memory optimization, and 60% sparsity while maintaining promising performance on zero-shot image classification and image-text retrieval tasks across 41 commonly used datasets. Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices. The model and code can be accessed from Hugging Face and GitHub.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 53 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

49 of 53 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
37 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  2. 2020Knowledge Distillation: A Surveycited by 1,603
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2019Learned Step Size Quantizationcited by 628
  13. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  14. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  15. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  16. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  17. 2019On the Efficacy of Knowledge Distillationcited by 381
  18. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  19. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  20. 2018Model compression via distillation and quantizationcited by 289
  21. 2021Post-Training Quantization for Vision Transformercited by 278
  22. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  23. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  24. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  25. 2016Trained Ternary Quantizationcited by 208
  26. 2016Ternary Weight Networkscited by 204
  27. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  28. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  29. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  30. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  31. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  32. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  33. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  34. 2016Ternary Neural Networks for Resource-Efficient AI Applicationscited by 33
  35. 2024Quantization without Tearscited by 25
  36. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  37. 2020FATNN: Fast and Accurate Ternary Neural Networkscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillationcited by 0
  2. 2026FTerViT: Fully Ternary Vision Transformercited by 0
  3. 2026On the Expressive Power of Weight Quantization in Large Language Modelscited by 0
  4. 2026Low-Rank Ternary Adaptation for Fine-Tuning Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  2. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  3. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
  4. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
  5. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  6. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  7. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  8. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations