# Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos

arXiv: https://arxiv.org/abs/2203.14104 · HTML: https://mlsanity.com/paper/2203.14104/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Timeception for Complex Action Recognition](https://mlsanity.com/paper/1812.01289/) (2018)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation](https://mlsanity.com/paper/2003.02824/) (2020)

## Built on by (most cited first, 30 of 50)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2309.04946/) (2023)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2212.02291/) (2022)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [How Much Temporal Long-Term Context is Needed for Action Segmentation?](https://mlsanity.com/paper/2308.11358/) (2023)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [SelfPromer: Self-Prompt Dehazing Transformers with Depth-Consistency](https://mlsanity.com/paper/2303.07033/) (2023)
- [Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models](https://mlsanity.com/paper/2312.06323/) (2023)
- [Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?](https://mlsanity.com/paper/2307.11978/) (2023)
- [Event-Guided Procedure Planning from Instructional Videos with Text Supervision](https://mlsanity.com/paper/2308.08885/) (2023)
- [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://mlsanity.com/paper/2303.04995/) (2023)
- [Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition](https://mlsanity.com/paper/2302.08102/) (2023)
- [Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning](https://mlsanity.com/paper/2310.00608/) (2023)
- [Recurrent Video Masked Autoencoders](https://mlsanity.com/paper/2512.13684/) (2025)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)
- [A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2312.05830/) (2023)
- [Glance and Focus: Memory Prompting for Multi-Event Video Question Answering](https://mlsanity.com/paper/2401.01529/) (2024)
- [Learning Point-Language Hierarchical Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2210.12513/) (2022)
- [Are current long-term video understanding datasets long-term?](https://mlsanity.com/paper/2308.11244/) (2023)
- [What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning](https://mlsanity.com/paper/2503.21055/) (2025)
- [SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08859/) (2023)
- [LEAP: LLM-Generation of Egocentric Action Programs](https://mlsanity.com/paper/2312.00055/) (2023)
- [Prompt-enhanced Hierarchical Transformer Elevating Cardiopulmonary Resuscitation Instruction via Temporal Action Segmentation](https://mlsanity.com/paper/2308.16552/) (2023)
- [ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation](https://mlsanity.com/paper/2412.04353/) (2024)
- [Temporal Segment Transformer for Action Segmentation](https://mlsanity.com/paper/2302.13074/) (2023)
- [Enhancing Transformer Backbone for Egocentric Video Action Segmentation](https://mlsanity.com/paper/2305.11365/) (2023)
- [HOI-aware Adaptive Network for Weakly-supervised Action Segmentation](https://mlsanity.com/paper/2604.26227/) (2026)
- [Proposal-based Temporal Action Localization with Point-level Supervision](https://mlsanity.com/paper/2310.05511/) (2023)

## Abstract (arXiv)

Action recognition models have shown a promising capability to classify human actions in short video clips. In a real scenario, multiple correlated human actions commonly occur in particular orders, forming semantically meaningful human activities. Conventional action recognition approaches focus on analyzing single actions. However, they fail to fully reason about the contextual relations between adjacent actions, which provide potential temporal logic for understanding long videos. In this paper, we propose a prompt-based framework, Bridge-Prompt (Br-Prompt), to model the semantics across adjacent actions, so that it simultaneously exploits both out-of-context and contextual information from a series of ordinal actions in instructional videos. More specifically, we reformulate the individual action labels as integrated text prompts for supervision, which bridge the gap between individual action semantics. The generated text prompts are paired with corresponding video clips, and together co-train the text encoder and the video encoder via a contrastive approach. The learned vision encoder has a stronger capability for ordinal-action-related downstream tasks, e.g. action segmentation and human activity recognition. We evaluate the performances of our approach on several video datasets: Georgia Tech Egocentric Activities (GTEA), 50Salads, and the Breakfast dataset. Br-Prompt achieves state-of-the-art on multiple benchmarks. Code is available at https://github.com/ttlmh/Bridge-Prompt