Training-free Online Video Step Grounding

Given a task and a set of steps composing it, video step grounding aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set with step-level annotations. This is costly obtain and may suffer from dataset imbalance for rare tasks. Moreover, the task is performed offline, i.e., processing the full video at once. This is a disadvantage, especially for environments requiring online decisions. In this work, we explore how to perform this task \textit{online} and \textit{without training}. We achieve this by using recent large multimodal models (LMMs), showing how their zero-shot capabilities are sufficient to outperform training-based models. We then propose a filtering mechanism that refines LMMs predictions by (i) adding a verification step on the model's output and (ii) modeling the step-transition probabilities via a dependency matrix and a transition one, both obtained through large language models. Experiments on three dataset show the efficacy of the approach.