# Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media

arXiv: https://arxiv.org/abs/2509.16811 · HTML: https://mlsanity.com/paper/2509.16811/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 21 of 21)

- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing](https://mlsanity.com/paper/2402.10294/) (2024)
- [DramaQA: Character-Centered Video Story Understanding with Hierarchical QA](https://mlsanity.com/paper/2005.03356/) (2020)
- [InstructVid2Vid: Controllable Video Editing with Natural Language Instructions](https://mlsanity.com/paper/2305.12328/) (2023)
- [AVscript: Accessible Video Editing with Audio-Visual Scripts](https://mlsanity.com/paper/2302.14117/) (2023)
- [ExpressEdit: Video Editing with Natural Language and Sketching](https://mlsanity.com/paper/2403.17693/) (2024)
- [VideoDiff: Human-AI Video Co-Creation with Alternatives](https://mlsanity.com/paper/2502.10190/) (2025)

## Built on by (most cited first, 6 of 6)

- [PPTArena: A Benchmark for PowerPoint Editing](https://mlsanity.com/paper/2512.03042/) (2025)
- [DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing](https://mlsanity.com/paper/2604.04875/) (2026)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [ELLMPEG: An Edge-based Agentic LLM Video Processing Tool](https://mlsanity.com/paper/2602.00028/) (2026)
- [BEAT: Rhythm-Elastic Alignment for Agentic Music-guided Movie Trailer Generation](https://mlsanity.com/paper/2605.27067/) (2026)
- [Unified Agentic Video Editing Across Levels of Complexity and Creativity](https://mlsanity.com/paper/2609.12769/) (2026)

## Abstract (arXiv)

Creators struggle to edit long-form, narrative-rich videos not because of UI complexity, but due to the cognitive demands of searching, storyboarding, and sequencing hours of footage. Existing transcript- or embedding-based methods fall short for creative workflows, as models struggle to track characters, infer motivations, and connect dispersed events. We present a prompt-driven, modular editing system that helps creators restructure multi-hour content through free-form prompts rather than timelines. At its core is a semantic indexing pipeline that builds a global narrative via temporal segmentation, guided memory compression, and cross-granularity fusion, producing interpretable traces of plot, dialogue, emotion, and context. Users receive cinematic edits while optionally refining transparent intermediate outputs. Evaluated on 400+ videos with expert ratings, QA, and preference studies, our system scales prompt-driven editing, preserves narrative coherence, and balances automation with creator control.