# Training-free Online Video Step Grounding

arXiv: https://arxiv.org/abs/2510.16989 · HTML: https://mlsanity.com/paper/2510.16989/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 19 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://mlsanity.com/paper/2409.04429/) (2024)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [Natural Language Video Localization: A Revisit in Span-based Question Answering Framework](https://mlsanity.com/paper/2102.13558/) (2021)
- [Towards A Better Metric for Text-to-Video Generation](https://mlsanity.com/paper/2401.07781/) (2024)
- [Revisiting the Role of Language Priors in Vision-Language Models](https://mlsanity.com/paper/2306.01879/) (2023)
- [Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos](https://mlsanity.com/paper/2503.13646/) (2025)
- [EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching](https://mlsanity.com/paper/2111.08919/) (2021)
- [Learning to Ground Instructional Articles in Videos through Narrations](https://mlsanity.com/paper/2306.03802/) (2023)
- [Video-Text Representation Learning via Differentiable Weak Temporal Alignment](https://mlsanity.com/paper/2203.16784/) (2022)
- [Multi-Sentence Grounding for Long-term Instructional Video](https://mlsanity.com/paper/2312.14055/) (2023)
- [Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment](https://mlsanity.com/paper/2409.16145/) (2024)
- [Can Text-to-Video Generation help Video-Language Alignment?](https://mlsanity.com/paper/2503.18507/) (2025)

## Abstract (arXiv)

Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.