# Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videos

arXiv: https://arxiv.org/abs/2503.08335 · HTML: https://mlsanity.com/paper/2503.08335/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Revealing Single Frame Bias for Video-and-Language Learning](https://mlsanity.com/paper/2206.03428/) (2022)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [Learning Language-Visual Embedding for Movie Understanding with Natural-Language](https://mlsanity.com/paper/1609.08124/) (2016)
- [Movie Summarization via Sparse Graph Construction](https://mlsanity.com/paper/2012.07536/) (2020)
- [Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videos](https://mlsanity.com/paper/2312.17117/) (2023)
- [PEEK: A Large Dataset of Learner Engagement with Educational Videos](https://mlsanity.com/paper/2109.03154/) (2021)
- [VLEngagement: A Dataset of Scientific Video Lectures for Evaluating Population-based Engagement](https://mlsanity.com/paper/2011.02273/) (2020)
- [Unsupervised Audio-Visual Lecture Segmentation](https://mlsanity.com/paper/2210.16644/) (2022)
- [Multi-modal News Understanding with Professionally Labelled Videos (ReutersViLNews)](https://mlsanity.com/paper/2401.12419/) (2024)

## Abstract (arXiv)

Learning multimodal video understanding typically relies on datasets comprising video clips paired with manually annotated captions. However, this becomes even more challenging when dealing with long-form videos, lasting from minutes to hours, in educational and news domains due to the need for more annotators with subject expertise. Hence, there arises a need for automated solutions. Recent advancements in Large Language Models (LLMs) promise to capture concise and informative content that allows the comprehension of entire videos by leveraging Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) technologies. ASR provides textual content from audio, while OCR extracts textual content from specific frames. This paper introduces a dataset comprising long-form lectures and news videos. We present baseline approaches to understand their limitations on this dataset and advocate for exploring prompt engineering techniques to comprehend long-form multimodal video datasets comprehensively.