# MM-AU:Towards Multimodal Understanding of Advertisement Videos

arXiv: https://arxiv.org/abs/2308.14052 · HTML: https://mlsanity.com/paper/2308.14052/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 108 references that are in the map.
- First posted on arXiv 27 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [Automatic Understanding of Image and Video Advertisements](https://mlsanity.com/paper/1707.03067/) (2017)
- [MovieCuts: A New Dataset and Benchmark for Cut Type Recognition](https://mlsanity.com/paper/2109.05569/) (2021)
- [The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing](https://mlsanity.com/paper/2207.09812/) (2022)
- [NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification](https://mlsanity.com/paper/1811.05014/) (2018)
- [Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset](https://mlsanity.com/paper/1912.05008/) (2019)
- [MovieCLIP: Visual Scene Recognition in Movies](https://mlsanity.com/paper/2210.11065/) (2022)
- [3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos](https://mlsanity.com/paper/2203.14456/) (2022)
- [Multimodal Content Analysis for Effective Advertisements on YouTube](https://mlsanity.com/paper/1709.03946/) (2017)
- [Story Understanding in Video Advertisements](https://mlsanity.com/paper/1807.11122/) (2018)

## Built on by (most cited first, 8 of 8)

- [Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning](https://mlsanity.com/paper/2506.07811/) (2025)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [TAU-R1: Visual Language Model for Traffic Anomaly Understanding](https://mlsanity.com/paper/2603.19098/) (2026)
- [Understanding Real-World Traffic Safety through RoadSafe365 Benchmark](https://mlsanity.com/paper/2602.07212/) (2026)
- [Leveraging multimodal explanatory annotations for video interpretation with Modality Specific Dataset](https://mlsanity.com/paper/2504.11232/) (2025)
- [Generating Animated Layouts as Structured Text Representations](https://mlsanity.com/paper/2505.00975/) (2025)
- [Toward Scalable Video Narration: A Training-free Approach Using Multimodal Large Language Models](https://mlsanity.com/paper/2507.17050/) (2025)
- [Beyond Single Videos: Benchmarking and Active Evidence Seeking for E-Commerce Cross-Video Reasoning](https://mlsanity.com/paper/2610.03099/) (2026)

## Abstract (arXiv)

Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures. The narrative structures of advertisements involve several elements like reasoning about the broad content (topic and the underlying message) and examining fine-grained details involving the transition of perceived tone due to the specific sequence of events and interaction among characters. In this work, to facilitate the understanding of advertisements along the three important dimensions of topic categorization, perceived tone transition, and social message detection, we introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts. Further, we demonstrate that leveraging signals from multiple modalities, including audio, video, and text, in multimodal transformer-based supervised models leads to improved performance compared to unimodal approaches.