# StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos

arXiv: https://arxiv.org/abs/2304.13265 · HTML: https://mlsanity.com/paper/2304.13265/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Procedure Planning in Instructional Videos](https://mlsanity.com/paper/1907.01172/) (2019)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision](https://mlsanity.com/paper/2205.02300/) (2022)
- [Going Deeper into First-Person Activity Recognition](https://mlsanity.com/paper/1605.03688/) (2016)
- [Representation Learning via Global Temporal Alignment and Cycle-Consistency](https://mlsanity.com/paper/2105.05217/) (2021)
- [Zero-Shot Anticipation for Instructional Activities](https://mlsanity.com/paper/1812.02501/) (2018)

## Built on by (most cited first, 30 of 37)

- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos](https://mlsanity.com/paper/2312.07322/) (2023)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [Video-Mined Task Graphs for Keystep Recognition in Instructional Videos](https://mlsanity.com/paper/2307.08763/) (2023)
- [Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos](https://mlsanity.com/paper/2406.01486/) (2024)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos](https://mlsanity.com/paper/2403.02782/) (2024)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [Multi-Sentence Grounding for Long-term Instructional Video](https://mlsanity.com/paper/2312.14055/) (2023)
- [MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video](https://mlsanity.com/paper/2408.03761/) (2024)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds](https://mlsanity.com/paper/2311.01444/) (2023)
- [HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos](https://mlsanity.com/paper/2505.12911/) (2025)
- [Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment](https://mlsanity.com/paper/2409.16145/) (2024)
- [EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos referring to Procedural Texts](https://mlsanity.com/paper/2410.05343/) (2024)
- [Learning Multiple Object States from Actions via Large Language Models](https://mlsanity.com/paper/2405.01090/) (2024)
- [GUIDE: A Guideline-Guided Dataset for Instructional Video Comprehension](https://mlsanity.com/paper/2406.18227/) (2024)
- [Procedure Learning via Regularized Gromov-Wasserstein Optimal Transport](https://mlsanity.com/paper/2507.15540/) (2025)
- [COM Kitchens: An Unedited Overhead-view Video Dataset as a Vision-Language Benchmark](https://mlsanity.com/paper/2408.02272/) (2024)
- [Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos](https://mlsanity.com/paper/2502.17753/) (2025)
- [CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planning](https://mlsanity.com/paper/2503.06637/) (2025)
- [ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios](https://mlsanity.com/paper/2603.09741/) (2026)
- [Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domain](https://mlsanity.com/paper/2312.11345/) (2023)
- [BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes](https://mlsanity.com/paper/2404.03161/) (2024)
- [ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos](https://mlsanity.com/paper/2411.15628/) (2024)
- [VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting](https://mlsanity.com/paper/2412.11621/) (2024)
- [REMAP: Regularized Matching and Partial Alignment of Video Embeddings](https://mlsanity.com/paper/2509.24382/) (2025)
- [ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency](https://mlsanity.com/paper/2606.25317/) (2026)
- [Efficient Pre-training for Localized Instruction Generation of Videos](https://mlsanity.com/paper/2311.15964/) (2023)

## Abstract (arXiv)

Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, with most of the video being irrelevant to the procedure. This motivates the need to temporally localize the instruction steps in such videos, i.e. the task called key-step localization. Traditional methods for key-step localization require video-level human annotations and thus do not scale to large datasets. In this work, we tackle the problem with no human supervision and introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. StepFormer is a transformer decoder that attends to the video with learnable queries, and produces a sequence of slots capturing the key-steps in the video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. In particular, we supervise our system with a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. We show that our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization by a large margin on three challenging benchmarks. Moreover, our model demonstrates an emergent property to solve zero-shot multi-step localization and outperforms all relevant baselines at this task.