Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment
also in 1 more
Abstract (arXiv)
Diffusion models have become the mainstream architecture for text-to-image generation, achieving remarkable progress in visual quality and prompt controllability. However, current inference pipelines generally lack interpretable semantic supervision and correction mechanisms throughout the denoising process. Most existing approaches rely solely on post-hoc scoring of the final image, prompt filtering, or heuristic resampling strategies-making them ineffective in providing actionable guidance for correcting the generative trajectory. As a result, models often suffer from object confusion, spatial errors, inaccurate counts, and missing semantic elements, severely compromising prompt-image alignment and image quality. To tackle these challenges, we propose MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD), a novel framework that, for the first time, introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference. PPAD performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps. The framework supports both inference-only and training-enhanced settings, and performs semantic correction at only extremely few diffusion steps, offering strong generality and scalability. Extensive experiments demonstrate PPAD's significant improvements.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 52 references that are in the map. First posted on arXiv 26 May 2025.
Builds on
50 of 52 references.
Closest ancestors · 1
Same direction, newest first
- 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs22 Jan
Other references · 49
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
- 2023Visual Instruction Tuningcited by 7,869
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2025Qwen2.5-VL Technical Reportcited by 5,587
37 more
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2021Variational Diffusion Modelscited by 1,153
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
- 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
- 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
- 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
- 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
- 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
- 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
- 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
- 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
- 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
- 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
- 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
- 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
- 2022Diffusion Probabilistic Models beat GANs on Medical Imagescited by 77
- 2022A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Imagescited by 74
- 2023PathLDM: Text conditioned Latent Diffusion Model for Histopathologycited by 52
- 2024Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflectioncited by 41
- 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 39
- 2023Generate Your Own Scotland: Satellite Image Generation Conditioned on Mapscited by 30
- 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 29
- 2023RSDiff: Remote Sensing Image Generation from Text Using Diffusion Modelcited by 29
- 2023ViT-DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysiscited by 15
Built on by
Citing papers per year: 2025: 1; 2026: 4.
5 papers.
- 2025LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Modelscited by 1
- 2026Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentationcited by 0
- 2026Evaluating Reasoning Fidelity in Visual Text Generationcited by 0
- 2026Training-Free Semantic Correction for Autoregressive Visual Modelscited by 0
- 2026EventOD: Event-Aware OD Flow Generation via LLM-Guided Semantic Modulationcited by 0
- 2026EventOD: Event-Aware OD Flow Generation via LLM-Guided Semantic Modulation26 Jun 2026
- 2026Training-Free Semantic Correction for Autoregressive Visual Models21 Jun 2026
- 2026Evaluating Reasoning Fidelity in Visual Text Generation3 Jun 2026
- 2026Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentation10 Mar 2026
- 2025LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models15 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2026MLLM-Guided Semantic Correction for Text-to-Video Generation
- 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
- 2024Dual Diffusion for Unified Image Generation and Understanding
- 2025D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
- 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
- 2024Improving Consistency in Diffusion Models for Image Super-Resolution
- 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
- 2026RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space