# CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

arXiv: https://arxiv.org/abs/2411.16828 · HTML: https://mlsanity.com/paper/2411.16828/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 25 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Built on by (most cited first, 22 of 22)

- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [MobileCLIP2: Improving Multi-Modal Reinforced Training](https://mlsanity.com/paper/2508.20691/) (2025)
- [When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.16203/) (2025)
- [OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning](https://mlsanity.com/paper/2505.04601/) (2025)
- [OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning](https://mlsanity.com/paper/2509.01644/) (2025)
- [Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning](https://mlsanity.com/paper/2505.24424/) (2025)
- [OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation](https://mlsanity.com/paper/2601.15369/) (2026)
- [SuperCLIP: CLIP with Simple Classification Supervision](https://mlsanity.com/paper/2512.14480/) (2025)
- [DiffCLIP: Differential Attention Meets CLIP](https://mlsanity.com/paper/2503.06626/) (2025)
- [Revisiting Multi-Task Visual Representation Learning](https://mlsanity.com/paper/2601.13886/) (2026)
- [Let ViT Speak: Generative Language-Image Pre-training](https://mlsanity.com/paper/2605.00809/) (2026)
- [Multimodal Representation Learning Conditioned on Semantic Relations](https://mlsanity.com/paper/2508.17497/) (2025)
- [Semantic search for 100M+ galaxy images using AI-generated captions](https://mlsanity.com/paper/2512.11982/) (2025)
- [Mining Contextualized Visual Associations from Images for Creativity Understanding](https://mlsanity.com/paper/2507.18915/) (2025)
- [ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering](https://mlsanity.com/paper/2509.16680/) (2025)
- [Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approach](https://mlsanity.com/paper/2511.05057/) (2025)
- [PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest](https://mlsanity.com/paper/2603.03544/) (2026)
- [No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models](https://mlsanity.com/paper/2603.25722/) (2026)
- [Investigating Adversarial Robustness of Multi-modal Large Language Models](https://mlsanity.com/paper/2606.03713/) (2026)
- [Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models](https://mlsanity.com/paper/2606.03730/) (2026)
- [ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP](https://mlsanity.com/paper/2606.26794/) (2026)
- [What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?](https://mlsanity.com/paper/2610.09700/) (2026)

## Abstract (arXiv)

Previous works show that noisy, web-crawled image-text pairs may limit vision-language pretraining like CLIP and propose learning with synthetic captions as a promising alternative. Our work continues this effort, introducing two simple yet effective designs to better leverage richly described synthetic captions. Firstly, by observing a strong inverse effect in learning with synthetic captions -- the short synthetic captions can generally lead to MUCH higher performance than full-length ones -- we therefore fed only partial synthetic captions to the text encoder. Secondly, we incorporate an autoregressive captioner to mimic the recaptioning process -- by conditioning on the paired image input and web-crawled text description, the captioner learns to predict the full-length synthetic caption generated by advanced MLLMs. Experiments show that our framework significantly improves zero-shot performance in cross-modal retrieval tasks, setting new SOTA results on MSCOCO and Flickr30K. Moreover, such trained vision encoders can enhance the visual capability of LLaVA, showing strong improvements on a range of MLLM benchmarks. Our project page is https://ucsc-vlaa.github.io/CLIPS/.