Abstract
Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at this https URL .
Abstract (translated)
视觉实体跟踪是人类的一种内在认知能力,然而对于视觉-语言模型(VLMs)来说,这仍然是一个关键的技术瓶颈。现有视频基准测试中所存在的视觉捷径常常掩盖了这种不足。我们引入了一个名为VET-Bench的合成诊断平台,该平台通过时空连续性来追踪视觉上完全相同的对象,从而展示了这一能力的重要性。实验表明,目前最先进的VLMs在VET-Bench上的表现接近随机水平,这揭示了一种基本限制:过度依赖静态帧级别的特征,并且无法随着时间维持实体表示。 我们进行理论分析并将其与状态跟踪问题联系起来,证明了基于固定深度变换器的VLMs由于表达能力的局限性,在没有中间监督的情况下追踪不可区分的对象时存在根本性的限制。为解决这一难题,我们提出了时空接地链式思维(Spatiotemporal Grounded Chain-of-Thought, SGCoT):通过生成物体轨迹作为显式的中间状态来解决问题。 利用Molmo2的物体跟踪能力,并在合成的文字数据上进行微调以实现对齐,从而激发SGCoT推理。我们的方法在VET-Bench上的准确率达到了90%以上,证明了VLMs可以在没有外部工具的情况下可靠地完成视频壳游戏任务。代码和数据可通过此链接获取。 该研究揭示了视觉-语言模型在处理时间连续性方面的能力局限,并提出了一种新颖的方法来解决这一问题,展示出潜在的改进路径和技术前景。
URL
https://arxiv.org/abs/2603.08436