Abstract
Generalizing across unknown targets is critical for open-world perception, yet existing 3D Multi-Object Tracking (3D MOT) pipelines remain limited by closed-set assumptions and ``semantic-blind'' heuristics. To address this, we propose Next-step Open-Vocabulary Autoregression (NOVA), an innovative paradigm that shifts 3D tracking from traditional fragmented distance-based matching toward generative spatio-temporal semantic modeling. NOVA reformulates 3D trajectories as structured spatio-temporal semantic sequences, enabling the simultaneous encoding of physical motion continuity and deep linguistic priors. By leveraging the autoregressive capabilities of Large Language Models (LLMs), we transform the tracking task into a principled process of next-step sequence completion. This mechanism allows the model to explicitly utilize the hierarchical structure of language space to resolve fine-grained semantic ambiguities and maintain identity consistency across complex long-range sequences through high-level commonsense reasoning. Extensive experiments on nuScenes, V2X-Seq-SPD, and KITTI demonstrate the superior performance of NOVA. Notably, on the nuScenes dataset, NOVA achieves an AMOTA of 22.41% for Novel categories, yielding a significant 20.21% absolute improvement over the baseline. These gains are realized through a compact 0.5B autoregressive model. Code will be available at this https URL.
Abstract (translated)
在开放世界的感知环境中，跨越未知目标进行泛化是至关重要的，然而现有的三维多对象跟踪（3D MOT）管道仍然受限于封闭集合假设和“无语义”的启发式方法。为了解决这个问题，我们提出了Next-step Open-Vocabulary Autoregression (NOVA)，这是一种创新的范例，将3D跟踪从传统的基于距离匹配的方法转变为生成性的时空语义建模。 NOVA重新定义了三维轨迹作为结构化的时空间语义序列，从而同时编码物理运动连续性和深层语言先验。通过利用大型语言模型（LLMs）的自回归能力，我们将追踪任务转化为一种原则上的下一步序列完成过程。这种方法使模型能够明确地使用语言空间中的层次结构来解决细粒度的语义模糊性，并通过高层次的常识推理在复杂的长距离序列中保持身份一致性。 在nuScenes、V2X-Seq-SPD和KITTI数据集上进行的大量实验表明，NOVA表现出色。特别值得注意的是，在nuScenes数据集中，对于新类别（Novel categories），NOVA实现了22.41%的AMOTA，比基线有了显著的20.21%绝对改进。这些改进是通过一个紧凑的0.5B自回归模型实现的。 代码将在这个链接提供：[此URL]
URL
https://arxiv.org/abs/2603.06254
