# Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding

arXiv: https://arxiv.org/abs/2411.17481 · HTML: https://mlsanity.com/paper/2411.17481/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 72 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Weakly Supervised Video Moment Retrieval From Text Queries](https://mlsanity.com/paper/1904.03282/) (2019)
- [Weakly-Supervised Video Moment Retrieval via Semantic Completion Network](https://mlsanity.com/paper/1911.08199/) (2019)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Hierarchical Local-Global Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2208.14882/) (2022)
- [Weakly Supervised Temporal Adjacent Network for Language Grounding](https://mlsanity.com/paper/2106.16136/) (2021)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model](https://mlsanity.com/paper/2307.12545/) (2023)
- [Video Moment Retrieval from Text Queries via Single Frame Annotation](https://mlsanity.com/paper/2204.09409/) (2022)
- [Progressive Localization Networks for Language-based Moment Localization](https://mlsanity.com/paper/2102.01282/) (2021)
- [Text-based Localization of Moments in a Video Corpus](https://mlsanity.com/paper/2008.08716/) (2020)
- [Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding](https://mlsanity.com/paper/2403.11463/) (2024)
- [End-to-End Dense Video Grounding via Parallel Regression](https://mlsanity.com/paper/2109.11265/) (2021)
- [Self-supervised Learning for Semi-supervised Temporal Language Grounding](https://mlsanity.com/paper/2109.11475/) (2021)
- [Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization](https://mlsanity.com/paper/2308.05648/) (2023)

## Built on by (most cited first, 2 of 2)

- [Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning](https://mlsanity.com/paper/2412.13543/) (2024)
- [Multi-QuAD: Multi-Level Quality-Adaptive Dynamic Network for Reliable Multimodal Classification](https://mlsanity.com/paper/2412.14489/) (2024)

## Abstract (arXiv)

Video Paragraph Grounding (VPG) aims to precisely locate the most appropriate moments within a video that are relevant to a given textual paragraph query. However, existing methods typically rely on large-scale annotated temporal labels and assume that the correspondence between videos and paragraphs is known. This is impractical in real-world applications, as constructing temporal labels requires significant labor costs, and the correspondence is often unknown. To address this issue, we propose a Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding method (DMR-JRG). In this method, retrieval and grounding tasks are mutually reinforced rather than being treated as separate issues. DMR-JRG mainly consists of two branches: a retrieval branch and a grounding branch. The retrieval branch uses inter-video contrastive learning to roughly align the global features of paragraphs and videos, reducing modality differences and constructing a coarse-grained feature space to break free from the need for correspondence between paragraphs and videos. Additionally, this coarse-grained feature space further facilitates the grounding branch in extracting fine-grained contextual representations. In the grounding branch, we achieve precise cross-modal matching and grounding by exploring the consistency between local, global, and temporal dimensions of video segments and textual paragraphs. By synergizing these dimensions, we construct a fine-grained feature space for video and textual features, greatly reducing the need for large-scale annotated temporal labels.