# Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward

arXiv: https://arxiv.org/abs/2209.12164 · HTML: https://mlsanity.com/paper/2209.12164/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 25 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Order Matters: Sequence to sequence for sets](https://mlsanity.com/paper/1511.06391/) (2015)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [Controlling Output Length in Neural Encoder-Decoders](https://mlsanity.com/paper/1609.09552/) (2016)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Less is More: Learning Highlight Detection from Video Duration](https://mlsanity.com/paper/1903.00859/) (2019)
- [A Graph-Based Framework to Bridge Movies and Synopses](https://mlsanity.com/paper/1910.11009/) (2019)
- [Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders](https://mlsanity.com/paper/1510.01442/) (2015)
- [Supervised Video Summarization via Multiple Feature Sets with Parallel Attention](https://mlsanity.com/paper/2104.11530/) (2021)
- [NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification](https://mlsanity.com/paper/1811.05014/) (2018)
- [Adaptive Video Highlight Detection by Learning from User History](https://mlsanity.com/paper/2007.09598/) (2020)
- [Adam^+: A Stochastic Method with Adaptive Variance Reduction](https://mlsanity.com/paper/2011.11985/) (2020)
- [Exact-K Recommendation via Maximal Clique Optimization](https://mlsanity.com/paper/1905.07089/) (2019)
- [From Trailers to Storylines: An Efficient Way to Learn from Movies](https://mlsanity.com/paper/1806.05341/) (2018)
- [Transforming Multi-Concept Attention into Video Summarization](https://mlsanity.com/paper/2006.01410/) (2020)
- [SUSiNet: See, Understand and Summarize it](https://mlsanity.com/paper/1812.00722/) (2018)
- [DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization](https://mlsanity.com/paper/2105.06441/) (2021)
- [Multi-modal Representation Learning for Video Advertisement Content Structuring](https://mlsanity.com/paper/2109.06637/) (2021)
- [Narrative Incoherence Detection](https://mlsanity.com/paper/2012.11157/) (2020)
- [Automatic Curation of Golf Highlights using Multimodal Excitement Features](https://mlsanity.com/paper/1707.07075/) (2017)

## Built on by (most cited first, 10 of 10)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Neural Combinatorial Optimization Algorithms for Solving Vehicle Routing Problems: A Comprehensive Survey with Perspectives](https://mlsanity.com/paper/2406.00415/) (2024)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [Tri2-plane: Thinking Head Avatar via Feature Pyramid](https://mlsanity.com/paper/2401.09386/) (2024)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment](https://mlsanity.com/paper/2412.00306/) (2024)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark](https://mlsanity.com/paper/2410.01180/) (2024)

## Abstract (arXiv)

Advertisement video editing aims to automatically edit advertising videos into shorter videos while retaining coherent content and crucial information conveyed by advertisers. It mainly contains two stages: video segmentation and segment assemblage. The existing method performs well at video segmentation stages but suffers from the problems of dependencies on extra cumbersome models and poor performance at the segment assemblage stage. To address these problems, we propose M-SAN (Multi-modal Segment Assemblage Network) which can perform efficient and coherent segment assemblage task end-to-end. It utilizes multi-modal representation extracted from the segments and follows the Encoder-Decoder Ptr-Net framework with the Attention mechanism. Importance-coherence reward is designed for training M-SAN. We experiment on the Ads-1k dataset with 1000+ videos under rich ad scenarios collected from advertisers. To evaluate the methods, we propose a unified metric, Imp-Coh@Time, which comprehensively assesses the importance, coherence, and duration of the outputs at the same time. Experimental results show that our method achieves better performance than random selection and the previous method on the metric. Ablation experiments further verify that multi-modal representation and importance-coherence reward significantly improve the performance. Ads-1k dataset is available at: https://github.com/yunlong10/Ads-1k