# Learning Vision from Models Rivals Learning Vision from Data

arXiv: https://arxiv.org/abs/2312.17742 · HTML: https://mlsanity.com/paper/2312.17742/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 89 of its 165 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 30 of 89)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)

## Built on by (most cited first, 30 of 71)

- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?](https://mlsanity.com/paper/2402.01832/) (2024)
- [Self-Correcting Self-Consuming Loops for Generative Model Training](https://mlsanity.com/paper/2402.07087/) (2024)
- [TIPS: Text-Image Pretraining with Spatial awareness](https://mlsanity.com/paper/2410.16512/) (2024)
- [Orient Anything V2: Unifying Orientation and Rotation Understanding](https://mlsanity.com/paper/2601.05573/) (2026)
- [The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better](https://mlsanity.com/paper/2406.05184/) (2024)
- [Minority-Focused Text-to-Image Generation via Prompt Optimization](https://mlsanity.com/paper/2410.07838/) (2024)
- [When Does Perceptual Alignment Benefit Vision Representations?](https://mlsanity.com/paper/2410.10817/) (2024)
- [Objective drives the consistency of representational similarity across datasets](https://mlsanity.com/paper/2411.05561/) (2024)
- [Improving the Scaling Laws of Synthetic Data with Deliberate Practice](https://mlsanity.com/paper/2502.15588/) (2025)
- [Understanding Bias in Large-Scale Visual Datasets](https://mlsanity.com/paper/2412.01876/) (2024)
- [A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.23398/) (2025)
- [Detecting Human Artifacts from Text-to-Image Models](https://mlsanity.com/paper/2411.13842/) (2024)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [Accessing Vision Foundation Models via ImageNet-1K](https://mlsanity.com/paper/2407.10366/) (2024)
- [Can OOD Object Detectors Learn from Foundation Models?](https://mlsanity.com/paper/2409.05162/) (2024)
- [How far can we go with ImageNet for Text-to-Image generation?](https://mlsanity.com/paper/2502.21318/) (2025)
- [Personalized Representation from Personalized Generation](https://mlsanity.com/paper/2412.16156/) (2024)
- [Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?](https://mlsanity.com/paper/2410.13523/) (2024)
- [Human-Aligned Image Models Improve Visual Decoding from the Brain](https://mlsanity.com/paper/2502.03081/) (2025)
- [Logic Unseen: Revealing the Logical Blindspots of Vision-Language Models](https://mlsanity.com/paper/2508.11317/) (2025)
- [Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems](https://mlsanity.com/paper/2508.12026/) (2025)
- [FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Models](https://mlsanity.com/paper/2401.10744/) (2024)
- [When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators](https://mlsanity.com/paper/2602.19946/) (2026)
- [Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?](https://mlsanity.com/paper/2405.18029/) (2024)
- [Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving](https://mlsanity.com/paper/2503.21449/) (2025)

## Abstract (arXiv)

We introduce SynCLR, a novel approach for learning visual representations exclusively from synthetic images and synthetic captions, without any real data. We synthesize a large dataset of image captions using LLMs, then use an off-the-shelf text-to-image model to generate multiple images corresponding to each synthetic caption. We perform visual representation learning on these synthetic images via contrastive learning, treating images sharing the same caption as positive pairs. The resulting representations transfer well to many downstream tasks, competing favorably with other general-purpose visual representation learners such as CLIP and DINO v2 in image classification tasks. Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR outperforms previous self-supervised methods by a significant margin, e.g., improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.