mlsanityData to 7 Oct 2026

MLLM-Guided Semantic Correction for Text-to-Video Generation

arXiv 2608.16513 ↗First posted 17 Aug 2026cs.CV cs.AIcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 42 of its 48 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

42 of 48 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing25 Nov
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  3. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Models12 Dec
  4. 2023MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing29 Nov
  5. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning21 Nov
  6. 2023LLM-grounded Video Diffusion Models29 Sep
  7. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  8. 2023ModelScope Text-to-Video Technical Report12 Aug
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  10. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  12. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
6 more
  1. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  2. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Video Diffusion Models7 Apr

Other references · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2023PaLM 2 Technical Reportcited by 1,154
12 more
  1. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2023TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessmentcited by 220
  4. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  5. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  6. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  7. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  8. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  9. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
  10. 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 29
  11. 2021Task-Specific Normalization for Continual Learning of Blind Image Quality Modelscited by 18
  12. 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
  2. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
  3. 2026Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation
  4. 2026KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
  5. 2024Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinement
  6. 2026Bernini: Latent Semantic Planning for Video Diffusion
  7. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
  8. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models