# Multi-modal Representation Learning for Video Advertisement Content Structuring

arXiv: https://arxiv.org/abs/2109.06637 · HTML: https://mlsanity.com/paper/2109.06637/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Fast Learning of Temporal Action Proposal via Dense Boundary Generator](https://mlsanity.com/paper/1911.04127/) (2019)

## Built on by (most cited first, 5 of 5)

- [Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward](https://mlsanity.com/paper/2209.12164/) (2022)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [Revisiting Pre-training in Audio-Visual Learning](https://mlsanity.com/paper/2302.03533/) (2023)
- [Generate the browsing process for short-video recommendation](https://mlsanity.com/paper/2504.08771/) (2025)
- [Decoding the Hook: A Multimodal LLM Framework for Analyzing the Hooking Period of Video Ads](https://mlsanity.com/paper/2602.22299/) (2026)

## Abstract (arXiv)

Video advertisement content structuring aims to segment a given video advertisement and label each segment on various dimensions, such as presentation form, scene, and style. Different from real-life videos, video advertisements contain sufficient and useful multi-modal content like caption and speech, which provides crucial video semantics and would enhance the structuring process. In this paper, we propose a multi-modal encoder to learn multi-modal representation from video advertisements by interacting between video-audio and text. Based on multi-modal representation, we then apply Boundary-Matching Network to generate temporal proposals. To make the proposals more accurate, we refine generated proposals by scene-guided alignment and re-ranking. Finally, we incorporate proposal located embeddings into the introduced multi-modal encoder to capture temporal relationships between local features of each proposal and global features of the whole video for classification. Experimental results show that our method achieves significantly improvement compared with several baselines and Rank 1 on the task of Multi-modal Ads Video Understanding in ACM Multimedia 2021 Grand Challenge. Ablation study further shows that leveraging multi-modal content like caption and speech in video advertisements significantly improve the performance.