# Weak Supervision and Referring Attention for Temporal-Textual Association Learning

arXiv: https://arxiv.org/abs/2006.11747 · HTML: https://mlsanity.com/paper/2006.11747/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/2310.05241/) (2023)
- [Weakly Supervised Relative Spatial Reasoning for Visual Question Answering](https://mlsanity.com/paper/2109.01934/) (2021)
- [RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2412.02402/) (2024)
- [Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization](https://mlsanity.com/paper/2308.05648/) (2023)
- [Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal](https://mlsanity.com/paper/2306.04502/) (2023)
- [ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision](https://mlsanity.com/paper/2204.06863/) (2022)
- [Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos](https://mlsanity.com/paper/2204.13548/) (2022)
- [AutoTVG: A New Vision-language Pre-training Paradigm for Temporal Video Grounding](https://mlsanity.com/paper/2406.07091/) (2024)

## Abstract (arXiv)

A system capturing the association between video frames and textual queries offer great potential for better video analysis. However, training such a system in a fully supervised way inevitably demands a meticulously curated video dataset with temporal-textual annotations. Therefore we provide a Weak-Supervised alternative with our proposed Referring Attention mechanism to learn temporal-textual association (dubbed WSRA). The weak supervision is simply a textual expression (e.g., short phrases or sentences) at video level, indicating this video contains relevant frames. The referring attention is our designed mechanism acting as a scoring function for grounding the given queries over frames temporally. It consists of multiple novel losses and sampling strategies for better training. The principle in our designed mechanism is to fully exploit 1) the weak supervision by considering informative and discriminative cues from intra-video segments anchored with the textual query, 2) multiple queries compared to the single video, and 3) cross-video visual similarities. We validate our WSRA through extensive experiments for temporally grounding by languages, demonstrating that it outperforms the state-of-the-art weakly-supervised methods notably.