# SKALD: Learning-Based Shot Assembly for Coherent Multi-Shot Video Creation

arXiv: https://arxiv.org/abs/2503.08010 · HTML: https://mlsanity.com/paper/2503.08010/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](https://mlsanity.com/paper/2207.02595/) (2022)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [Quality Assessment of In-the-Wild Videos](https://mlsanity.com/paper/1908.00375/) (2019)
- [Learning Debiased Representation via Disentangled Feature Augmentation](https://mlsanity.com/paper/2107.01372/) (2021)
- [CAT: Cross Attention in Vision Transformer](https://mlsanity.com/paper/2106.05786/) (2021)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval](https://mlsanity.com/paper/2308.07648/) (2023)
- [Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations](https://mlsanity.com/paper/2204.03382/) (2022)
- [StarVQA: Space-Time Attention for Video Quality Assessment](https://mlsanity.com/paper/2108.09635/) (2021)
- [StableVQA: A Deep No-Reference Quality Assessment Model for Video Stability](https://mlsanity.com/paper/2308.04904/) (2023)
- [Transcript to Video: Efficient Clip Sequencing from Texts](https://mlsanity.com/paper/2107.11851/) (2021)
- [V3C - a Research Video Collection](https://mlsanity.com/paper/1810.04401/) (2018)
- [StarVQA+: Co-training Space-Time Attention for Video Quality Assessment](https://mlsanity.com/paper/2306.12298/) (2023)

## Built on by (most cited first, 3 of 3)

- [OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory](https://mlsanity.com/paper/2512.07802/) (2025)
- [ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search](https://mlsanity.com/paper/2601.23232/) (2026)
- [MSEditor: Toward Consistent Multi-Shot Video Editing](https://mlsanity.com/paper/2608.17559/) (2026)

## Abstract (arXiv)

We present SKALD, a multi-shot video assembly method that constructs coherent video sequences from candidate shots with minimal reliance on text. Central to our approach is the Learned Clip Assembly (LCA) score, a learning-based metric that measures temporal and semantic relationships between shots to quantify narrative coherence. We tackle the exponential complexity of combining multiple shots with an efficient beam-search algorithm guided by the LCA score. To train our model effectively with limited human annotations, we propose two tasks for the LCA encoder: Shot Coherence Learning, which uses contrastive learning to distinguish coherent and incoherent sequences, and Feature Regression, which converts these learned representations into a real-valued coherence score. We develop two variants: a base SKALD model that relies solely on visual coherence and SKALD-text, which integrates auxiliary text information when available. Experiments on the VSPD and our curated MSV3C datasets show that SKALD achieves an improvement of up to 48.6% in IoU and a 43% speedup over the state-of-the-art methods. A user study further validates our approach, with 45% of participants favoring SKALD-assembled videos, compared to 22% preferring text-based assembly methods.