# OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

arXiv: https://arxiv.org/abs/2604.11102 · HTML: https://mlsanity.com/paper/2604.11102/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 31)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos](https://mlsanity.com/paper/2411.19772/) (2024)
- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2506.15220/) (2025)
- [ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts](https://mlsanity.com/paper/2507.20939/) (2025)
- [Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research](https://mlsanity.com/paper/1503.01070/) (2015)
- [AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration](https://mlsanity.com/paper/2510.10395/) (2025)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models](https://mlsanity.com/paper/2601.19267/) (2026)
- [TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions](https://mlsanity.com/paper/2602.08711/) (2026)
- [Movie101v2: Improved Movie Narration Benchmark](https://mlsanity.com/paper/2404.13370/) (2024)
- [ARC-Chapter: Structuring Hour-Long Videos into Navigable Chapters and Hierarchical Summaries](https://mlsanity.com/paper/2511.14349/) (2025)

## Built on by (most cited first, 6 of 6)

- [OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains](https://mlsanity.com/paper/2606.14702/) (2026)
- [CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales](https://mlsanity.com/paper/2606.21949/) (2026)
- [AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning](https://mlsanity.com/paper/2607.12820/) (2026)
- [Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search](https://mlsanity.com/paper/2605.08762/) (2026)
- [OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning](https://mlsanity.com/paper/2610.12458/) (2026)
- [PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary](https://mlsanity.com/paper/2610.16066/) (2026)

## Abstract (arXiv)

Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.