# What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing

arXiv: https://arxiv.org/abs/2605.20795 · HTML: https://mlsanity.com/paper/2605.20795/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)
- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 19 of 19)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [UniVideo: Unified Understanding, Generation, and Editing for Videos](https://mlsanity.com/paper/2510.08377/) (2025)
- [EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods](https://mlsanity.com/paper/2310.02426/) (2023)
- [ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement](https://mlsanity.com/paper/2504.01934/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance](https://mlsanity.com/paper/2603.02175/) (2026)
- [InstructX: Towards Unified Visual Editing with MLLM Guidance](https://mlsanity.com/paper/2510.08485/) (2025)
- [IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment](https://mlsanity.com/paper/2510.11647/) (2025)
- [OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning](https://mlsanity.com/paper/2603.24458/) (2026)
- [VINO: A Unified Visual Generator with Interleaved OmniModal Context](https://mlsanity.com/paper/2601.02358/) (2026)
- [Query-Kontext: An Unified Multimodal Model for Image Generation and Editing](https://mlsanity.com/paper/2509.26641/) (2025)

## Abstract (arXiv)

Flow matching based video generative models have been increasingly relying on prepended Vision-Language Models (VLMs) to handle complex, instruction-based video editing. The prevailing assumption underlying this paradigm is that a connector module can seamlessly align the VLM's rich multi-modal reasoning with the original text embedding space of DiTs. However, we hypothesize that this alignment acts as a severe semantic bottleneck, degrading fine-grained structural variables. Verifying this is challenging, as end-to-end evaluations conflate alignment failures with generation errors, and natural datasets lack disentangled annotations. To rigorously investigate this, we propose a controlled data processing pipeline based on video composition that results in TRACE-Edit, a diagnostic dataset focusing on relation-based editing. Leveraging this dataset, we propose a comprehensive diagnostic protocol to analyze two important designs of meta-query and connector in the existing video editing models. Systematic evaluation of four representative model cases reveals that fine-grained structural semantics can be severely degraded during alignment. Our findings overturn the assumption of lossless semantic transfer, identifying the VLM-to-DiT alignment as a major bottleneck and providing a new diagnostic foundation for future multi-modal alignment architectures.