# KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

arXiv: https://arxiv.org/abs/2607.14202 · HTML: https://mlsanity.com/paper/2607.14202/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 150 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 29 of 29)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [SkyReels-V2: Infinite-length Film Generative Model](https://mlsanity.com/paper/2504.13074/) (2025)
- [T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](https://mlsanity.com/paper/2407.14505/) (2024)
- [Seedance 2.0: Advancing Video Generation for World Complexity](https://mlsanity.com/paper/2604.14148/) (2026)
- [ElasticFace: Elastic Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/2109.09416/) (2021)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [Kling-Omni Technical Report](https://mlsanity.com/paper/2512.16776/) (2025)
- [Mavors: Multi-granularity Video Representation for Multimodal Large Language Model](https://mlsanity.com/paper/2504.10068/) (2025)
- [VABench: A Comprehensive Benchmark for Audio-Video Generation](https://mlsanity.com/paper/2512.09299/) (2025)
- [AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI](https://mlsanity.com/paper/2401.01651/) (2024)
- [VideoMemory: Toward Consistent Video Generation via Memory Integration](https://mlsanity.com/paper/2601.03655/) (2026)
- [Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model](https://mlsanity.com/paper/2603.21986/) (2026)
- [T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.21094/) (2025)
- [AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation](https://mlsanity.com/paper/2604.08540/) (2026)
- [MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation](https://mlsanity.com/paper/2602.23969/) (2026)
- [UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark](https://mlsanity.com/paper/2509.24427/) (2025)
- [MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation](https://mlsanity.com/paper/2604.23789/) (2026)
- [VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning](https://mlsanity.com/paper/2510.08555/) (2025)
- [DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation](https://mlsanity.com/paper/2512.21252/) (2025)
- [A Mixed Diet Makes DINO An Omnivorous Vision Encoder](https://mlsanity.com/paper/2602.24181/) (2026)
- [SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control](https://mlsanity.com/paper/2605.27891/) (2026)

## Built on by (most cited first, 1 of 1)

- [RefCaptioner: Multi-Reference Image-Grounded Video Captioning](https://mlsanity.com/paper/2607.28509/) (2026)

## Abstract (arXiv)

Video generation increasingly relies on keyframe-based workflows, where creators specify a sequence of reference images to guide generation. Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribed keyframes while maintaining overall video quality. We present KeyFrame-Compass, the first comprehensive benchmark for evaluating keyframe-conditioned video generation. The benchmark contains 386 carefully curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities, enabling controlled analysis under diverse generation settings. We further introduce an automated evaluation framework that jointly measures keyframe execution and overall video quality. Specifically, we decompose keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models. Experiments on nine representative video generation systems reveal several fundamental limitations. Current models exhibit a clear trade-off between faithful keyframe execution and natural video synthesis. Their performance further degrades as keyframe constraints become denser and most open-source models also fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences.