Abstract
Object tracking is an important step in robotics and reautonomous driving pipelines, which has to generalize to previously unseen and complex objects. Existing high-performing methods often rely on pre-captured object views to build explicit reference models, which restricts them to a fixed set of known objects. However, such reference models can struggle with visually complex appearance, reducing the quality of tracking. In this work, we introduce an object tracking method based on light field images that does not depend on a pre-trained model, while being robust to complex visual behavior, such as reflections. We extract semantic and geometric features from light field inputs using vision foundation models and convert them into view-dependent Gaussian splats. These splats serve as a unified object representation, supporting differentiable rendering and pose optimization. We further introduce a light field object tracking dataset containing challenging reflective objects with precise ground truth poses. Experiments demonstrate that our method is competitive with state-of-the-art model-based trackers in these difficult cases, paving the way toward universal object tracking in robotic systems. Code/data available at this https URL.
Abstract (translated)
对象跟踪是机器人和自动驾驶流水线中的一个重要步骤，需要对以前未见过的复杂物体进行泛化。现有的高性能方法通常依赖于预捕获的对象视图来构建显式的参考模型，这限制了它们只能处理一组已知的对象。然而，这样的参考模型在面对视觉上复杂的外观时会遇到困难，从而降低了跟踪质量。在这项工作中，我们介绍了一种基于光场图像的对象跟踪方法，该方法无需依赖预训练的模型，并且对诸如反射等复杂视觉行为具有鲁棒性。我们利用视觉基础模型从光场输入中提取语义和几何特征，并将其转换为视图相关的高斯点（Gaussian splats）。这些点作为统一的对象表示形式，支持可微渲染和姿态优化。此外，我们还引入了一个包含挑战性的反射物体的光场对象跟踪数据集，并提供了精确的姿态地面实况。实验表明，在这些困难的情况下，我们的方法与最先进的模型基于的方法具有竞争力，为机器人系统中的通用对象跟踪铺平了道路。代码和数据可在该链接获取。
URL
https://arxiv.org/abs/2512.13007
