mlsanityData to 7 Oct 2026

Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models

arXiv 2409.10695 ↗First posted 16 Sep 2024cs.CV cs.AI cs.GRcited by 113

Not on the map. Why?

Cited by 57 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Playground v3 (PGv3), our latest text-to-image model that achieves state-of-the-art (SoTA) performance across multiple testing benchmarks, excels in graphic design abilities and introduces new capabilities. Unlike traditional text-to-image generative models that rely on pre-trained language models like T5 or CLIP text encoders, our approach fully integrates Large Language Models (LLMs) with a novel structure that leverages text conditions exclusively from a decoder-only LLM. Additionally, to enhance image captioning quality-we developed an in-house captioner, capable of generating captions with varying levels of detail, enriching the diversity of text structures. We also introduce a new benchmark CapsBench to evaluate detailed image captioning performance. Experimental results demonstrate that PGv3 excels in text prompt adherence, complex reasoning, and accurate text rendering. User preference studies indicate the super-human graphic design ability of our model for common design applications, such as stickers, posters, and logo designs. Furthermore, PGv3 introduces new capabilities, including precise RGB color control and robust multilingual understanding.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

40 of 49 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2022Flow Matching for Generative Modelingcited by 5,426
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
28 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  10. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  11. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  12. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  13. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  14. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  15. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  16. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  17. 2022Analyzing Transformers in Embedding Spacecited by 131
  18. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  19. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  20. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  21. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 51
  22. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  23. 2024MoMA: Multimodal LLM Adapter for Fast Personalized Image Generationcited by 35
  24. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generationcited by 30
  25. 2024DiffusionAgent: Navigating Expert Models for Agentic Image Generationcited by 27
  26. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  27. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddingscited by 18
  28. 2021QACE: Asking Questions to Evaluate an Image Captioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 67; 2026: 23.

113 papers.

Sort:

By area:

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  4. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  5. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  6. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  7. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  8. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  9. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  10. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  11. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  12. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
38 more
  1. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  2. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  3. 2025Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Modelscited by 61
  4. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  5. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 55
  6. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  7. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  8. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  9. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 44
  10. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  11. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  12. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  13. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  14. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  15. 2025Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Imagescited by 35
  16. 2025InfiniteYou: Flexible Photo Recrafting While Preserving Your Identitycited by 34
  17. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  18. 20241.58-bit FLUXcited by 31
  19. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  20. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  21. 2025SenseFlow: Scaling Distribution Matching for Flow-based Text-to-Image Distillationcited by 29
  22. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  23. 2024SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Trainingcited by 27
  24. 2024Training-free Regional Prompting for Diffusion Transformerscited by 26
  25. 2024EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLMcited by 25
  26. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 22
  27. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  28. 2025TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flowscited by 17
  29. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  30. 2025AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generationcited by 16
  31. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  32. 2025UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Datasetcited by 16
  33. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  34. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  35. 2025LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesiscited by 15
  36. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesiscited by 15
  37. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 13
  38. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignment
  2. 2025M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
  3. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
  4. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  5. 2025Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
  6. 2026Learning to Rank Caption Chains for Video-Text Alignment
  7. 2023Aligning Text-to-Image Models using Human Feedback
  8. 2025Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Models