# Coherent Multi-Sentence Video Description with Variable Level of Detail

arXiv: https://arxiv.org/abs/1403.6173 · HTML: https://mlsanity.com/paper/1403.6173/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2014.

## Built on by (most cited first, 30 of 45)

- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [MovieChat+: Question-aware Sparse Memory for Long Video Question Answering](https://mlsanity.com/paper/2404.17176/) (2024)
- [SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries](https://mlsanity.com/paper/2304.04565/) (2023)
- [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021)
- [NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory](https://mlsanity.com/paper/2301.00746/) (2023)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [InfiniBench: A Benchmark for Large Multi-Modal Models in Long-Form Movies and TV Shows](https://mlsanity.com/paper/2406.19875/) (2024)
- [SpotEM: Efficient Video Search for Episodic Memory](https://mlsanity.com/paper/2306.15850/) (2023)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Hierarchical3D Adapters for Long Video-to-text Summarization](https://mlsanity.com/paper/2210.04829/) (2022)
- [Weakly-Supervised Temporal Article Grounding](https://mlsanity.com/paper/2210.12444/) (2022)
- [Learning Streaming Video Representation via Multitask Training](https://mlsanity.com/paper/2504.20041/) (2025)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [MICap: A Unified Model for Identity-aware Movie Descriptions](https://mlsanity.com/paper/2405.11483/) (2024)
- [GUI Action Narrator: Where and When Did That Action Take Place?](https://mlsanity.com/paper/2406.13719/) (2024)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding](https://mlsanity.com/paper/2601.15016/) (2026)
- [Discriminative Latent Semantic Graph for Video Captioning](https://mlsanity.com/paper/2108.03662/) (2021)
- [Multi Sentence Description of Complex Manipulation Action Videos](https://mlsanity.com/paper/2311.07285/) (2023)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding](https://mlsanity.com/paper/2508.04546/) (2025)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)
- [Temporally Grounding Instructional Diagrams in Unconstrained Videos](https://mlsanity.com/paper/2407.12066/) (2024)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)

## Abstract (arXiv)

Humans can easily describe what they see in a coherent way and at varying level of detail. However, existing approaches for automatic video description are mainly focused on single sentence generation and produce descriptions at a fixed level of detail. In this paper, we address both of these limitations: for a variable level of detail we produce coherent multi-sentence descriptions of complex videos. We follow a two-step approach where we first learn to predict a semantic representation (SR) from video and then generate natural language descriptions from the SR. To produce consistent multi-sentence descriptions, we model across-sentence consistency at the level of the SR by enforcing a consistent topic. We also contribute both to the visual recognition of objects proposing a hand-centric approach as well as to the robust generation of sentences using a word lattice. Human judges rate our multi-sentence descriptions as more readable, correct, and relevant than related work. To understand the difference between more detailed and shorter descriptions, we collect and analyze a video description corpus of three levels of detail.