# VidParse: Online Parsing of Egocentric Procedures Like a Pro

arXiv: https://arxiv.org/abs/2608.27562 · HTML: https://mlsanity.com/paper/2608.27562/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [VideoGraph: Recognizing Minutes-Long Human Activities in Videos](https://mlsanity.com/paper/1905.05143/) (2019)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2111.14799/) (2021)
- [Test-Time Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2404.05426/) (2024)
- [LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervision](https://mlsanity.com/paper/2304.07647/) (2023)
- [Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment](https://mlsanity.com/paper/2403.19225/) (2024)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [Joint Self-Supervised Video Alignment and Action Segmentation](https://mlsanity.com/paper/2503.16832/) (2025)
- [Trajectory-aligned Space-time Tokens for Few-shot Action Recognition](https://mlsanity.com/paper/2407.18249/) (2024)
- [Activity Grammars for Temporal Action Segmentation](https://mlsanity.com/paper/2312.04266/) (2023)
- [Online Temporal Action Localization with Memory-Augmented Transformer](https://mlsanity.com/paper/2408.02957/) (2024)
- [OnlineTAS: An Online Baseline for Temporal Action Segmentation](https://mlsanity.com/paper/2411.01122/) (2024)
- [HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos](https://mlsanity.com/paper/2505.12911/) (2025)
- [Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition](https://mlsanity.com/paper/2508.03695/) (2025)
- [Online Generic Event Boundary Detection](https://mlsanity.com/paper/2510.06855/) (2025)
- [Fine-grained Dynamic Network for Generic Event Boundary Detection](https://mlsanity.com/paper/2407.04274/) (2024)
- [Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models](https://mlsanity.com/paper/2501.13795/) (2025)

## Abstract (arXiv)

Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.