# What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?

arXiv: https://arxiv.org/abs/2610.09700 · HTML: https://mlsanity.com/paper/2610.09700/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 81 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [CyCLIP: Cyclic Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2205.14459/) (2022)
- [EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters](https://mlsanity.com/paper/2402.04252/) (2024)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [SILC: Improving Vision Language Pretraining with Self-Distillation](https://mlsanity.com/paper/2310.13355/) (2023)
- [Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding](https://mlsanity.com/paper/2306.08832/) (2023)
- [CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions](https://mlsanity.com/paper/2411.16828/) (2024)
- [MLLMs-Augmented Visual-Language Representation Learning](https://mlsanity.com/paper/2311.18765/) (2023)
- [Manifold Mixup: Better Representations by Interpolating Hidden States](https://mlsanity.com/paper/1806.05236/) (2018)
- [A Review on Discriminative Self-supervised Learning Methods in Computer Vision](https://mlsanity.com/paper/2405.04969/) (2024)
- [SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning](https://mlsanity.com/paper/2410.02401/) (2024)
- [ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives](https://mlsanity.com/paper/2609.01041/) (2026)

## Abstract (arXiv)

Synthetic hard negatives generated in the representation space have proven effective for unimodal self-supervised learning, but transferring this idea to vision-language pretraining is not straightforward. We analyze six representation-space synthesis strategies and identify two failure modes in their transfer to vision-language pretraining: cross-modal constructions that produce overly easy negatives or pull them toward the query, and intra-modal constructions that incorporate the matched positive. We also observe logit-scale saturation when training with synthetic hard negatives and a learnable temperature, and find that fixing the temperature improves downstream performance. Using this geometric analysis we propose SNAP, which generates intra-modal hard negatives that never involve the positive from either modality, avoiding both failure modes entirely. SNAP is model-agnostic, requires no external generative models, and adds less than 10% training time overhead. Evaluated on top of CLIP and FLIP across multiple architectures and datasets, SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation.