Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
Not on the map. Why?
Abstract (arXiv)
Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 56 of its 73 references that are in the map. First posted on arXiv 3 Mar 2026.
Builds on
56 of 73 references.
References · 56
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
- 2013Describing Textures in the Wildcited by 2,110
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
- 2019Natural Adversarial Examplescited by 1,401
- 2025DINOv3cited by 1,297
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
- 2016Deep Networks with Stochastic Depthcited by 973
- 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
- 2024Autoregressive Image Generation without Vector Quantizationcited by 644
- 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
- 2017Large Batch Training of Convolutional Networkscited by 529
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
- 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
- 2022Scaling Language-Image Pre-training via Maskingcited by 317
- 2019A Baseline for Few-Shot Image Classificationcited by 298
- 2025Diffusion Transformers with Representation Autoencoderscited by 275
- 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
- 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
- 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
- 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
- 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
- 20234M: Massively Multimodal Masked Modelingcited by 118
- 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
- 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
- 2023ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classificationcited by 33
- 2023Learning Visual Representations via Language-Guided Samplingcited by 32
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.