Paper Reading AI Learner

Can Vision-Language Models Solve the Shell Game?

2026-03-09 14:33:25
Tiedong Liu, Wee Sun Lee

Abstract

Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at this https URL .

Abstract (translated)

视觉实体跟踪是人类的一种内在认知能力,然而对于视觉-语言模型(VLMs)来说,这仍然是一个关键的技术瓶颈。现有视频基准测试中所存在的视觉捷径常常掩盖了这种不足。我们引入了一个名为VET-Bench的合成诊断平台,该平台通过时空连续性来追踪视觉上完全相同的对象,从而展示了这一能力的重要性。实验表明,目前最先进的VLMs在VET-Bench上的表现接近随机水平,这揭示了一种基本限制:过度依赖静态帧级别的特征,并且无法随着时间维持实体表示。 我们进行理论分析并将其与状态跟踪问题联系起来,证明了基于固定深度变换器的VLMs由于表达能力的局限性,在没有中间监督的情况下追踪不可区分的对象时存在根本性的限制。为解决这一难题,我们提出了时空接地链式思维(Spatiotemporal Grounded Chain-of-Thought, SGCoT):通过生成物体轨迹作为显式的中间状态来解决问题。 利用Molmo2的物体跟踪能力,并在合成的文字数据上进行微调以实现对齐,从而激发SGCoT推理。我们的方法在VET-Bench上的准确率达到了90%以上,证明了VLMs可以在没有外部工具的情况下可靠地完成视频壳游戏任务。代码和数据可通过此链接获取。 该研究揭示了视觉-语言模型在处理时间连续性方面的能力局限,并提出了一种新颖的方法来解决这一问题,展示出潜在的改进路径和技术前景。

URL

https://arxiv.org/abs/2603.08436

PDF

https://arxiv.org/pdf/2603.08436.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot