mlsanityData to 7 Oct 2026

Using Multimodal Foundation Models and Clustering for Improved Style Ambiguity Loss

arXiv 2407.12009 ↗First posted 20 Jun 2024cs.CV cs.AIcited by 1

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Teaching text-to-image models to be creative involves using style ambiguity loss, which requires a pretrained classifier. In this work, we explore a new form of the style ambiguity training objective, used to approximate creativity, that does not require training a classifier or even a labeled dataset. We then train a diffusion model to maximize style ambiguity to imbue the diffusion model with creativity and find our new methods improve upon the traditional method, based on automated metrics for human judgment, while still maintaining creativity and novelty.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 22 of its 45 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

22 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Diffusion-based Image Translation using Disentangled Style and Content Representation30 Sep

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2015Continuous control with deep reinforcement learningcited by 3,279
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
9 more
  1. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  4. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  5. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  6. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  7. 2016A guide to convolution arithmetic for deep learningcited by 182
  8. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  9. 2017CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Normscited by 96

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024I Spy With My Little Eye: A Minimum Cost Multicut Investigation of Dataset Framescited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Style Ambiguity Loss Using CLIP
  2. 2024WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylization
  3. 2022Language Does More Than Describe: On The Lack Of Figurative Speech in Text-To-Image Models
  4. 2025Enhancing Creative Generation on Stable Diffusion-based Models
  5. 2025StyleBlend: Enhancing Style-Specific Content Creation in Text-to-Image Diffusion Models
  6. 2025The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models
  7. 2025Distribution-Conditional Generation: From Class Distribution to Creative Generation
  8. 2024AttnMod: Attention-Based New Art Styles