# PLOT-TAL: Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localization

arXiv: https://arxiv.org/abs/2403.18915 · HTML: https://mlsanity.com/paper/2403.18915/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://mlsanity.com/paper/2112.09583/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [PLOT: Prompt Learning with Optimal Transport for Vision-Language Models](https://mlsanity.com/paper/2210.01253/) (2022)
- [A Survey on Optimal Transport for Machine Learning: Theory and Applications](https://mlsanity.com/paper/2106.01963/) (2021)
- [An Efficient Spatio-Temporal Pyramid Transformer for Action Detection](https://mlsanity.com/paper/2207.10448/) (2022)
- [Few-Shot Temporal Action Localization with Query Adaptive Transformer](https://mlsanity.com/paper/2110.10552/) (2021)
- [Temporal Action Localization with Enhanced Instant Discriminability](https://mlsanity.com/paper/2309.05590/) (2023)
- [Localizing the Common Action Among a Few Videos](https://mlsanity.com/paper/2008.05826/) (2020)
- [Revisiting Few-shot Activity Detection with Class Similarity Control](https://mlsanity.com/paper/2004.00137/) (2020)
- [Multi-Modal Few-Shot Temporal Action Detection](https://mlsanity.com/paper/2211.14905/) (2022)

## Built on by (most cited first, 3 of 3)

- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localization](https://mlsanity.com/paper/2504.13460/) (2025)
- [Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy](https://mlsanity.com/paper/2501.18509/) (2025)

## Abstract (arXiv)

Few-shot temporal action localization (TAL) methods that adapt large models via single-prompt tuning often fail to produce precise temporal boundaries. This stems from the model learning a non-discriminative mean representation of an action from sparse data, which compromises generalization. We address this by proposing a new paradigm based on multi-prompt ensembles, where a set of diverse, learnable prompts for each action is encouraged to specialize on compositional sub-events. To enforce this specialization, we introduce PLOT-TAL, a framework that leverages Optimal Transport (OT) to find a globally optimal alignment between the prompt ensemble and the video's temporal features. Our method establishes a new state-of-the-art on the challenging few-shot benchmarks of THUMOS'14 and EPIC-Kitchens, without requiring complex meta-learning. The significant performance gains, particularly at high IoU thresholds, validate our hypothesis and demonstrate the superiority of learning distributed, compositional representations for precise temporal localization.