# PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning

arXiv: https://arxiv.org/abs/2506.16082 · HTML: https://mlsanity.com/paper/2506.16082/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 68 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)

## Abstract (arXiv)

Dense video captioning is a challenging task that aims to localize and caption multiple events in an untrimmed video. Recent studies mainly follow the transformer-based architecture to jointly perform the two sub-tasks, i.e., event localization and caption generation, in an end-to-end manner. Based on the general philosophy of detection transformer, these methods implicitly learn the event locations and event semantics, which requires a large amount of training data and limits the model's performance in practice. In this paper, we propose a novel dense video captioning framework, named PR-DETR, which injects the explicit position and relation prior into the detection transformer to improve the localization accuracy and caption quality, simultaneously. On the one hand, we first generate a set of position-anchored queries to provide the scene-specific position and semantic information about potential events as position prior, which serves as the initial event search regions to eliminate the implausible event proposals. On the other hand, we further design an event relation encoder to explicitly calculate the relationship between event boundaries as relation prior to guide the event interaction to improve the semantic coherence of the captions. Extensive ablation studies are conducted to verify the effectiveness of the position and relation prior. Experimental results also show the competitive performance of our method on ActivityNet Captions and YouCook2 datasets.