mlsanityData to 14 Oct 2026

Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion

arXiv 2505.20053 ↗First posted 26 May 2025cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

Diffusion models have become the mainstream architecture for text-to-image generation, achieving remarkable progress in visual quality and prompt controllability. However, current inference pipelines generally lack interpretable semantic supervision and correction mechanisms throughout the denoising process. Most existing approaches rely solely on post-hoc scoring of the final image, prompt filtering, or heuristic resampling strategies-making them ineffective in providing actionable guidance for correcting the generative trajectory. As a result, models often suffer from object confusion, spatial errors, inaccurate counts, and missing semantic elements, severely compromising prompt-image alignment and image quality. To tackle these challenges, we propose MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD), a novel framework that, for the first time, introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference. PPAD performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps. The framework supports both inference-only and training-enhanced settings, and performs semantic correction at only extremely few diffusion steps, offering strong generality and scalability. Extensive experiments demonstrate PPAD's significant improvements.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 52 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

50 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs22 Jan

Other references · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2025Qwen2.5-VL Technical Reportcited by 5,587
37 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  12. 2021Variational Diffusion Modelscited by 1,153
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  15. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  16. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  17. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  18. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  21. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  22. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  23. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  24. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  25. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  26. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  27. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  28. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  29. 2022Diffusion Probabilistic Models beat GANs on Medical Imagescited by 77
  30. 2022A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Imagescited by 74
  31. 2023PathLDM: Text conditioned Latent Diffusion Model for Histopathologycited by 52
  32. 2024Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflectioncited by 41
  33. 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 39
  34. 2023Generate Your Own Scotland: Satellite Image Generation Conditioned on Mapscited by 30
  35. 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 29
  36. 2023RSDiff: Remote Sensing Image Generation from Text Using Diffusion Modelcited by 29
  37. 2023ViT-DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysiscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026MLLM-Guided Semantic Correction for Text-to-Video Generation
  2. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  3. 2024Dual Diffusion for Unified Image Generation and Understanding
  4. 2025D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
  5. 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
  6. 2024Improving Consistency in Diffusion Models for Image Super-Resolution
  7. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
  8. 2026RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space