MLLM-Guided Semantic Correction for Text-to-Video Generation
On the map: Diffusion models → Video generation
Abstract (arXiv)
Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 42 of its 48 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
42 of 48 references.
Closest ancestors · 18
Same direction, newest first
- 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing25 Nov
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
- 2023FreeInit: Bridging Initialization Gap in Video Diffusion Models12 Dec
- 2023MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing29 Nov
- 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning21 Nov
- 2023LLM-grounded Video Diffusion Models29 Sep
- 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
- 2023ModelScope Text-to-Video Technical Report12 Aug
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
- 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
- 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
6 more
- 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
- 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
- 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
- 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
- 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
- 2022Video Diffusion Models7 Apr
Other references · 24
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2023GPT-4 Technical Reportcited by 18,132
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2023PaLM 2 Technical Reportcited by 1,154
12 more
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
- 2023TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessmentcited by 220
- 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
- 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
- 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
- 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
- 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
- 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
- 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 29
- 2021Task-Specific Normalization for Continual Learning of Blind Image Quality Modelscited by 18
- 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
- 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
- 2026Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation
- 2026KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
- 2024Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinement
- 2026Bernini: Latent Semantic Planning for Video Diffusion
- 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
- 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models