# Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization

arXiv: https://arxiv.org/abs/2203.16800 · HTML: https://mlsanity.com/paper/2203.16800/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 75 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 30 of 53)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization](https://mlsanity.com/paper/1704.04232/) (2017)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [Weakly Supervised Action Localization by Sparse Temporal Pooling Network](https://mlsanity.com/paper/1712.05080/) (2017)
- [W-TALC: Weakly-supervised Temporal Activity Localization and Classification](https://mlsanity.com/paper/1807.10418/) (2018)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [AutoLoc: Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1807.08333/) (2018)
- [Background Suppression Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1911.09963/) (2019)
- [CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning](https://mlsanity.com/paper/2103.16392/) (2021)
- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [SF-Net: Single-Frame Supervision for Temporal Action Localization](https://mlsanity.com/paper/2003.06845/) (2020)
- [Weakly-Supervised Action Localization by Generative Attention Modeling](https://mlsanity.com/paper/2003.12424/) (2020)
- [Differentiable Dynamic Programming for Structured Prediction and Attention](https://mlsanity.com/paper/1802.03676/) (2018)
- [Weakly-supervised Action Localization with Background Modeling](https://mlsanity.com/paper/1908.06552/) (2019)
- [3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization](https://mlsanity.com/paper/1908.08216/) (2019)
- [Weakly Supervised Action Labeling in Videos Under Ordering Constraints](https://mlsanity.com/paper/1407.1208/) (2014)
- [Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning](https://mlsanity.com/paper/2004.00163/) (2020)
- [A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2101.00545/) (2021)

## Built on by (most cited first, 30 of 33)

- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment](https://mlsanity.com/paper/2405.06887/) (2024)
- [Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2305.17861/) (2023)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Multi-modal Prompting for Low-Shot Temporal Action Localization](https://mlsanity.com/paper/2303.11732/) (2023)
- [Improving Audio-Visual Video Parsing with Pseudo Visual Labels](https://mlsanity.com/paper/2303.02344/) (2023)
- [Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space](https://mlsanity.com/paper/2206.11895/) (2022)
- [Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.09335/) (2022)
- [Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2307.02041/) (2023)
- [Boosting Weakly-Supervised Temporal Action Localization with Text Information](https://mlsanity.com/paper/2305.00607/) (2023)
- [Revisiting Foreground and Background Separation in Weakly-supervised Temporal Action Localization: A Clustering-based Approach](https://mlsanity.com/paper/2312.14138/) (2023)
- [Generalized Uncertainty-Based Evidential Fusion with Hybrid Multi-Head Attention for Weak-Supervised Temporal Action Localization](https://mlsanity.com/paper/2412.19418/) (2024)
- [DDG-Net: Discriminability-Driven Graph Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2307.16415/) (2023)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [M2DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer](https://mlsanity.com/paper/2305.08877/) (2023)
- [Forcing the Whole Video as Background: An Adversarial Learning Strategy for Weakly Temporal Action Localization](https://mlsanity.com/paper/2207.06659/) (2022)
- [Slow Motion Matters: A Slow Motion Enhanced Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2211.11324/) (2022)
- [Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling](https://mlsanity.com/paper/2308.09946/) (2023)
- [Cross-Video Contextual Knowledge Exploration and Exploitation for Ambiguity Reduction in Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2308.12609/) (2023)
- [Full-Stage Pseudo Label Quality Enhancement for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2407.08971/) (2024)
- [Weakly-Supervised Temporal Action Localization with Bidirectional Semantic Consistency Constraint](https://mlsanity.com/paper/2304.12616/) (2023)
- [Proposal-based Temporal Action Localization with Point-level Supervision](https://mlsanity.com/paper/2310.05511/) (2023)
- [Weak Supervision with Arbitrary Single Frame for Micro- and Macro-expression Spotting](https://mlsanity.com/paper/2403.14240/) (2024)
- [DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification](https://mlsanity.com/paper/2506.12585/) (2025)
- [Beyond Label Semantics: Language-Guided Action Anatomy for Few-shot Action Recognition](https://mlsanity.com/paper/2507.16287/) (2025)
- [Sub-action Prototype Learning for Point-level Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2309.09060/) (2023)
- [POTLoc: Pseudo-Label Oriented Transformer for Point-Supervised Temporal Action Localization](https://mlsanity.com/paper/2310.13585/) (2023)
- [Video-Specific Query-Key Attention Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2305.04186/) (2023)
- [ADM-Loc: Actionness Distribution Modeling for Point-supervised Temporal Action Localization](https://mlsanity.com/paper/2311.15916/) (2023)
- [Computer User Interface Understanding. A New Dataset and a Learning Framework](https://mlsanity.com/paper/2403.10170/) (2024)

## Abstract (arXiv)

We target at the task of weakly-supervised action localization (WSAL), where only video-level action labels are available during model training. Despite the recent progress, existing methods mainly embrace a localization-by-classification paradigm and overlook the fruitful fine-grained temporal distinctions between video sequences, thus suffering from severe ambiguity in classification learning and classification-to-localization adaption. This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in WSAL and helps identify coherent action instances. Specifically, under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting, where the first one considers the relations of various action/background proposals by using match, insert, and delete operators and the second one mines the longest common subsequences between two videos. Both contrasting modules can enhance each other and jointly enjoy the merits of discriminative action-background separation and alleviated task gap between classification and localization. Extensive experiments show that our method achieves state-of-the-art performance on two popular benchmarks. Our code is available at https://github.com/MengyuanChen21/CVPR2022-FTCL.