# Jointly Localizing and Describing Events for Dense Video Captioning

arXiv: https://arxiv.org/abs/1804.08274 · HTML: https://mlsanity.com/paper/1804.08274/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2018.

## Built on by (most cited first, 30 of 60)

- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [UnIVAL: Unified Model for Image, Video, Audio and Language Tasks](https://mlsanity.com/paper/2307.16184/) (2023)
- [VideoXum: Cross-modal Visual and Textural Summarization of Videos](https://mlsanity.com/paper/2303.12060/) (2023)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [End-to-end Dense Video Captioning as Sequence Generation](https://mlsanity.com/paper/2204.08121/) (2022)
- [VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2211.15103/) (2022)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval](https://mlsanity.com/paper/2204.00486/) (2022)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising](https://mlsanity.com/paper/2112.07515/) (2021)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network](https://mlsanity.com/paper/2101.11562/) (2021)
- [Unifying Event Detection and Captioning as Sequence Generation via Pre-Training](https://mlsanity.com/paper/2207.08625/) (2022)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [ARGUS: Hallucination and Omission Evaluation in Video-LLMs](https://mlsanity.com/paper/2506.07371/) (2025)
- [DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement](https://mlsanity.com/paper/2404.02755/) (2024)
- [DistinctAD: Distinctive Audio Description Generation in Contexts](https://mlsanity.com/paper/2411.18180/) (2024)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics](https://mlsanity.com/paper/2108.08217/) (2021)

## Abstract (arXiv)

Automatically describing a video with natural language is regarded as a fundamental challenge in computer vision. The problem nevertheless is not trivial especially when a video contains multiple events to be worthy of mention, which often happens in real videos. A valid question is how to temporally localize and then describe events, which is known as "dense video captioning." In this paper, we present a novel framework for dense video captioning that unifies the localization of temporal event proposals and sentence generation of each proposal, by jointly training them in an end-to-end manner. To combine these two worlds, we integrate a new design, namely descriptiveness regression, into a single shot detection structure to infer the descriptive complexity of each detected proposal via sentence generation. This in turn adjusts the temporal locations of each event proposal. Our model differs from existing dense video captioning methods since we propose a joint and global optimization of detection and captioning, and the framework uniquely capitalizes on an attribute-augmented video captioning architecture. Extensive experiments are conducted on ActivityNet Captions dataset and our framework shows clear improvements when compared to the state-of-the-art techniques. More remarkably, we obtain a new record: METEOR of 12.96% on ActivityNet Captions official test set.