# Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks

arXiv: https://arxiv.org/abs/2502.18326 · HTML: https://mlsanity.com/paper/2502.18326/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2025.

## Built on by (most cited first, 7 of 7)

- [Does Data Scaling Lead to Visual Compositional Generalization?](https://mlsanity.com/paper/2507.07102/) (2025)
- [What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives](https://mlsanity.com/paper/2510.03075/) (2025)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [When Do Diffusion Models learn to Generate Multiple Objects?](https://mlsanity.com/paper/2605.00273/) (2026)
- [Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models](https://mlsanity.com/paper/2507.08000/) (2025)
- [Are Object-Centric Representations Better At Compositional Generalization?](https://mlsanity.com/paper/2602.16689/) (2026)

## Abstract (arXiv)

We investigate the success conditions for compositional generalization of CLIP models on real-world data through performance prediction. Prior work shows that CLIP requires exponentially more pretraining data for linear performance gains on individual concepts. This sample-inefficient scaling could be mitigated if CLIP systematically understood new inputs as compositions of learned components, allowing rare observation to be mapped to common concepts. To explore CLIP's compositional generalization ability, we filter retrieval corpora for samples with object combinations not present in the pretraining corpus. We show that CLIP's performance on these samples can be accurately predicted from the pretraining frequencies of individual objects. Our findings demonstrate that CLIP learns to disentangle objects observed in its pretraining data and can recompose them straightforwardly. Additionally, we are the first to show how this ability scales with pretraining data. For data curation in practice, our results suggest that balancing object occurrences improves generalization, which should benefit CLIP's efficiency and accuracy without scaling data volume.