# Move Forward and Tell: A Progressive Generator of Video Descriptions

arXiv: https://arxiv.org/abs/1807.10018 · HTML: https://mlsanity.com/paper/1807.10018/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2018.

## Built on by (most cited first, 30 of 35)

- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Text with Knowledge Graph Augmented Transformer for Video Captioning](https://mlsanity.com/paper/2303.12423/) (2023)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2211.15103/) (2022)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2206.12972/) (2022)
- [Unifying Event Detection and Captioning as Sequence Generation via Pre-Training](https://mlsanity.com/paper/2207.08625/) (2022)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos](https://mlsanity.com/paper/2303.06378/) (2023)
- [End-to-End Video Question-Answer Generation with Generator-Pretester Network](https://mlsanity.com/paper/2101.01447/) (2021)
- [Weakly Supervised Dense Video Captioning via Jointly Usage of Knowledge Distillation and Cross-modal Matching](https://mlsanity.com/paper/2105.08252/) (2021)
- [Dense Video Captioning Using Unsupervised Semantic Information](https://mlsanity.com/paper/2112.08455/) (2021)
- [Contextual Explainable Video Representation: Human Perception-based Understanding](https://mlsanity.com/paper/2212.06206/) (2022)
- [Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization](https://mlsanity.com/paper/2408.14547/) (2024)
- [FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation](https://mlsanity.com/paper/2411.15413/) (2024)
- [Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers](https://mlsanity.com/paper/2108.02147/) (2021)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality](https://mlsanity.com/paper/2403.19221/) (2024)
- [DVCFlow: Modeling Information Flow Towards Human-like Video Captioning](https://mlsanity.com/paper/2111.10146/) (2021)
- [Multi-modal News Understanding with Professionally Labelled Videos (ReutersViLNews)](https://mlsanity.com/paper/2401.12419/) (2024)
- [Live Video Captioning](https://mlsanity.com/paper/2406.14206/) (2024)
- [METEOR Guided Divergence for Video Captioning](https://mlsanity.com/paper/2212.10690/) (2022)
- [Implicit and Explicit Commonsense for Multi-sentence Video Captioning](https://mlsanity.com/paper/2303.07545/) (2023)

## Abstract (arXiv)

We present an efficient framework that can generate a coherent paragraph to describe a given video. Previous works on video captioning usually focus on video clips. They typically treat an entire video as a whole and generate the caption conditioned on a single embedding. On the contrary, we consider videos with rich temporal structures and aim to generate paragraph descriptions that can preserve the story flow while being coherent and concise. Towards this goal, we propose a new approach, which produces a descriptive paragraph by assembling temporally localized descriptions. Given a video, it selects a sequence of distinctive clips and generates sentences thereon in a coherent manner. Particularly, the selection of clips and the production of sentences are done jointly and progressively driven by a recurrent network -- what to describe next depends on what have been said before. Here, the recurrent network is learned via self-critical sequence training with both sentence-level and paragraph-level rewards. On the ActivityNet Captions dataset, our method demonstrated the capability of generating high-quality paragraph descriptions for videos. Compared to those by other methods, the descriptions produced by our method are often more relevant, more coherent, and more concise.