# Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning

arXiv: https://arxiv.org/abs/2603.11439 · HTML: https://mlsanity.com/paper/2603.11439/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Selective Structured State-Spaces for Long-Form Video Understanding](https://mlsanity.com/paper/2303.14526/) (2023)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning](https://mlsanity.com/paper/1804.00100/) (2018)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Repulsion Loss: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1711.07752/) (2017)
- [Jointly Localizing and Describing Events for Dense Video Captioning](https://mlsanity.com/paper/1804.08274/) (2018)
- [Streamlined Dense Video Captioning](https://mlsanity.com/paper/1904.03870/) (2019)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning](https://mlsanity.com/paper/1909.09944/) (2019)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives](https://mlsanity.com/paper/2308.02066/) (2023)
- [Video Captioning with Guidance of Multimodal Latent Topics](https://mlsanity.com/paper/1708.09667/) (2017)
- [Self-Sufficient Framework for Continuous Sign Language Recognition](https://mlsanity.com/paper/2303.11771/) (2023)

## Built on by (most cited first, 1 of 1)

- [Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning](https://mlsanity.com/paper/2609.04183/) (2026)

## Abstract (arXiv)

Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.