Abstract
Understanding fine-grained actions and accurately localizing their corresponding actors in space and time are fundamental capabilities for advancing next-generation AI systems, including embodied agents, autonomous platforms, and human-AI interaction frameworks. Despite recent progress in video understanding, existing methods predominantly address either coarse-grained action recognition or generic object tracking, thereby overlooking the challenge of jointly detecting and tracking multiple objects according to their actions while grounding them temporally. To address this gap, we introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos based on natural language descriptions of their actions. To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs, covering a diverse range of objects, actions, and real-world scenes. We further propose SVAGFormer, a baseline framework that adapts state of the art vision language models for joint spatial and temporal grounding, and introduce SVAGEval, a standardized evaluation toolkit for fair and reproducible benchmarking. Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes, underscoring the need for more advanced reasoning over fine-grained object-action interactions in long videos.
Abstract (translated)
理解细微动作并准确地在时间和空间上定位这些动作的执行者是推进下一代人工智能系统（包括具身代理、自主平台和人机交互框架）的基本能力。尽管近期视频理解方面取得了进展，现有的方法主要集中在粗粒度的动作识别或通用对象跟踪上，从而忽视了同时检测和根据动作追踪多个对象并将其时间定位的问题。为解决这一缺口，我们提出了时空视频行动接地（SVAG），这是一个要求模型基于自然语言描述的动作来同时检测、跟踪并在视频中临时定位所有相关对象的新任务。 为了支持这项任务，我们构建了SVAG-Bench，一个大规模基准测试集，包括688个视频、19,590条标注记录和903个独特的动词，涵盖了多样化的物体、动作和现实场景。此外，我们提出了SVAGFormer，一种基于最先进的视觉语言模型的基线框架，用于联合空间和时间定位，并引入了SVAGEval，一个标准化的评估工具包，用于公平和可重复的基准测试。 实证结果显示，现有的模型在SVAG任务上表现不佳，尤其是在密集或复杂的场景中，这强调了对长时间视频中的细粒度对象-动作交互进行更高级推理的需求。
URL
https://arxiv.org/abs/2510.13016
