Abstract
Human perception for effective object tracking in a 2D video stream arises from the implicit use of prior 3D knowledge combined with semantic reasoning. In contrast, most generic object tracking (GOT) methods primarily rely on 2D features of the target and its surroundings while neglecting 3D geometric cues, which makes them susceptible to partial occlusion, distractors, and variations in geometry and appearance. To address this limitation, we introduce GOT-Edit, an online cross-modality model editing approach that integrates geometry-aware cues into a generic object tracker from a 2D video stream. Our approach leverages features from a pre-trained Visual Geometry Grounded Transformer to enable geometric cue inference from only a few 2D images. To tackle the challenge of seamlessly combining geometry and semantics, GOT-Edit performs online model editing with null-space constrained updates that incorporate geometric information while preserving semantic discrimination, yielding consistently better performance across diverse scenarios. Extensive experiments on multiple GOT benchmarks demonstrate that GOT-Edit achieves superior robustness and accuracy, particularly under occlusion and clutter, establishing a new paradigm for combining 2D semantics with 3D geometric reasoning for generic object tracking.
Abstract (translated)
人类在二维视频流中进行有效物体跟踪的能力来源于对隐含的三维知识和语义推理的运用。相比之下，大多数通用对象跟踪（GOT）方法主要依赖于目标及其周围环境的二维特征，并忽略几何线索，这使它们容易受到部分遮挡、干扰物以及形状和外观变化的影响。为了克服这一局限性，我们引入了GOT-Edit，这是一种在线跨模态模型编辑方法，它将几何感知线索融入到从二维视频流中获取的通用对象跟踪器中。我们的方法利用预训练的视觉几何基础变换器（Visual Geometry Grounded Transformer）中的特征，仅通过少数几幅二维图像就能推断出几何线索。 为了应对无缝结合几何和语义的挑战，GOT-Edit采用在线模型编辑技术，并使用零空间约束更新来整合几何信息同时保持语义区分性，从而在各种场景中获得更优性能。在多个GOT基准测试中的广泛实验表明，GOT-Edit实现了卓越的鲁棒性和准确性，特别是在遮挡和杂乱环境下表现尤为突出，这标志着结合二维语义与三维几何推理的新范式用于通用对象跟踪。
URL
https://arxiv.org/abs/2602.08550
