# SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning

arXiv: https://arxiv.org/abs/2603.05437 · HTML: https://mlsanity.com/paper/2603.05437/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning](https://mlsanity.com/paper/1804.00100/) (2018)
- [Jointly Localizing and Describing Events for Dense Video Captioning](https://mlsanity.com/paper/1804.08274/) (2018)
- [Weakly Supervised Dense Event Captioning in Videos](https://mlsanity.com/paper/1812.03849/) (2018)
- [Streamlined Dense Video Captioning](https://mlsanity.com/paper/1904.03870/) (2019)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [Video Captioning with Guidance of Multimodal Latent Topics](https://mlsanity.com/paper/1708.09667/) (2017)
- [IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning](https://mlsanity.com/paper/2409.18046/) (2024)
- [Exploiting Auxiliary Caption for Video Grounding](https://mlsanity.com/paper/2301.05997/) (2023)
- [Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning](https://mlsanity.com/paper/2412.12791/) (2024)
- [HiCM2: Hierarchical Compact Memory Modeling for Dense Video Captioning](https://mlsanity.com/paper/2412.14585/) (2024)
- [VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness](https://mlsanity.com/paper/2503.16406/) (2025)
- [Weakly Supervised Dense Video Captioning via Jointly Usage of Knowledge Distillation and Cross-modal Matching](https://mlsanity.com/paper/2105.08252/) (2021)
- [ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion](https://mlsanity.com/paper/2510.25818/) (2025)
- [SIDA: Synthetic Image Driven Zero-shot Domain Adaptation](https://mlsanity.com/paper/2507.18632/) (2025)
- [Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning](https://mlsanity.com/paper/2509.04602/) (2025)
- [SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioning](https://mlsanity.com/paper/2507.18616/) (2025)
- [CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation](https://mlsanity.com/paper/2507.18750/) (2025)

## Built on by (most cited first, 1 of 1)

- [Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning](https://mlsanity.com/paper/2609.04183/) (2026)

## Abstract (arXiv)

Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.